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FOREWORD 


This  document  is  Volume  7  in  a  series  produced  by  the  Army 
Research  Institute  for  the  Behavioral  and  Social  Sciences  (ARI)  and  the 
Project  Manager  for  Training  Devices  (PM  TRADE).  The  series  consists 
of  10  related  documents  that  present  guidance  for  combat  and  training 
systems  developers,  Including  Army  Materiel  Command  (AMC)  and  other 
materiel  developers  and  entities,  Training  and  Doctrine  Commemd 
(TRADOC)  Combat  Developers  and  Training  Developers,  and  contractor 
organizations  involved  in  system  development  or  developing  techno¬ 
logical  thrust  areas  under  Independent  research  and  development  (IR&D) 
programs . 

This  series  of  documents  includes  guidelines  and  procedures  that 
support  the  effective  consideration,  definition,  development,  and 
integration  of  embedded  training  (ET)  capabilities  for  existing  and 
developmental  systems.  The  10  documents  share  the  general  title  of 
Implementing  Embedded  Training  (ET^.  with  specific,  descriptive  sub¬ 
titles  for  each  document.  They  arc  as  follows: 


1 .  Volume  1:  Overview  presents  an  overall  view  of  the  guidance 
documents  and  their  contents,  purposes,  and  applications, 
including  a  discussion  of  the  following: 

a.  what  the  total  training  system  concept,  including  embedded 
training,  is; 

b.  how  training  systems  must  develop  within  more  general 
processes  of  materiel  system  development; 

c.  how  embedded  training  must  affect  this  relationship;  and 

d.  what  the  content  and  uses  of  the  remaining  documents  in 
the  series  are,  as  well  as  their  relationships  to  the 
training  systems  development  and  acquisition  processes, 
and  how  to  use  them. 

2.  Volume  2;  ET  as  a  System  Alternative  provides  guidelines  for 

the  initial  decision  on  whether  ET  should  be  further  con¬ 
sidered  as  a  training  system  alternative  for  a  given  materiel 
system.  It  also  Includes  guidance  on  considering  ET  as  an 
alternative  for  systems  under  product  improvement  or  modifica¬ 
tion,  after  fielding.  _ _ 

n  For 

3.  Volume  3;  The  Role  of  ET  in  the  Training  System  Concent  con-  ,ti~ 

tains  guidance  for  the  early  estimation  of  training  system 
requirements  and  the  potential  allocation  of  such  requirements  oj 
to  ET.  ,'„loa 


V 


i"' . . 

1  BV . . 

I  Distribution/ 


Availability  Co<5«^ 
iAvall  and/or 


Dlst 


Special 


□  □ 


A.  Volume  4;  Identifying  ET  Renulrements  presents  procedures  for 
defining  ET  requirements  (ETRs)  at  both  initial  levels  (i.e., 
before  initiating  systems  development)  and  for  revising  and 
updating  initial  ETRs  during  system  design  and  development. 

5.  Volume  5:  Designing  the  ET  Component  contains  analytic  proce¬ 
dures  and  guidance  for  designing  an  ET  component  concept  for  a 
materiel  system,  based  on  specified  ETRs. 

6.  Volume  6:  Integrating  ET  vlth  the  Prime  System  discusses  con¬ 
siderations,  guidance,  and  "lessons  learned"  about  factors 
that  influence  the  effective  integration  of  ET  Into  materiel 
systems. 

7.  Volume  7:  ET  Test  and  Evaluation  presents  guidance  for  defin¬ 
ing  the  aspects  of  the  ET  component  (test  Issues)  to  be 
addressed  in  prototype  and  full-scale  system  testing. 

8.  Volvimo  8;  Incorporating  ET  Into  Unit  Training  provides  guid¬ 
ance  for  integrating  ET  considerations  and  Information  into 
unit  training  documentation  and  practice. 

9.  Volume  9;  Logistics  Implications  presents  helpful  information 
on  key  logistics  Issues  that  should  be  addressed  In  the  con¬ 
text  of  ET  integration  vith  prime  Item  systems. 

1 0 .  Volume  10:  Intetrratlng  ET  into  Acquisition  Documenta t i on 
provides  guidance  on  developing  the  necessary  documentation 
for,  and  specification  of,  an  ET  Component  of  a  prime  item 
during  the  Army’s  systems  development  and  acquisition  process. 
This  document  examines  the  Life  Cycle  System  Management  Model 
(LCSMM)  and  the  Army  Streamlined  Acquisition  Process  (ASAP) 
and  describes  vhere  and  how  to  include  ET  considerations  in 
the  associated  documentation.  It  also  describes  vhere  and  hov 
to  use  the  other  volumes  in  the  ET  guidelines  series  to  gener¬ 
ate  the  Information  required  for  the  acquisition  documenta¬ 
tion,  and  provides  guidance  in  preparing  a  contract  Statement 
of  Work  for  an  ET  Component  to  a  prime  item  system. 


(ju>  ■  — 


WILLIAM  KARROLETTI 


Deputy  Project  Manager 


EDGAR  M.  JOHNSON 
Technical  Director 
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INTRODUCTION 


The  purpose  of  this  volume  is  to  provide  a  general  approach  and 
guidance  for  Test  and  Evaluation  (T&E)  of  the  embedded  training  (ET) 
subsystem  during  system  development.  System  development  which  includes 
integrated  ET  is  relatively  new  to  the  Army  acquisition  process.  Many 
existing  regulations  and  specifications  have  not,  as  yet,  been  updated 
to  reflect  the  ET  development  process  integrated  with  the  system  devel¬ 
opment  process.  The  series  of  ET  guideline  documents,  of  which  this 
document  is  a  part,  structures  and  procedural izes  the  process.  These 
ET  development  procedures  may  become  formalized  and  included  in  acqui¬ 
sition  documentation  at  some  later  point  in  time.  This  volinne 
discusses  the  major  issues  of  ET  T&E. 

ET  imposes  a  number  of  new  and  significant  requirements  on  each  of 
the  stages  of  system  development,  especially  in  the  early  stages  when 
system  hardware,  software,  and  operational  concepts  are  being  formu¬ 
lated  and  refined.  The  training  T&E  requirements  imposed  on  the  system 
development  environment  are  particularly  unique  from  the  perspective  of 
training  developers.  Training  T&E  requirements  assume  an  urgency  that 
is  not  present  v^en  ET  is  not  a  part  of  the  system  development  process. 
T&E  for  integrated  ET  is  in  its  infancy.  Little  experience  has  been 
gained  to  guide  and  structure  the  T&E  process  because  at  this  time  no 
Army  procurements  Chat  included  ET  requirements  based  on  defined 
system-specific  training  requirements  have  been  brought  through  the 
production  phase.  It  is  expected  Chat  the  basic  guidelines  offered 
here  may,  in  the  future,  be  expanded  through  applying  lessons  learned. 


Overview 


The  overall  purpose  of  this  guideline  is  to  present  guidance  on 
how  to  incorporate  unique  ET  considerations  into  ongoing  T&E  procedures 
for  the  developing  system.  This  documei.C  attempts  to  highlight  those 
added  T&E  considerations  \diich  will  be  beneficial— even  essential— to 
the  overall  process  which  integrates  ET  (along  with  other  functional 
components)  into  an  efficient  and  cost-effective  end-item.  Elements 
that  are  required  for  general  T&E  are  not  covered  here  unless  they  are 
particularly  critical  to  the  ET  T&E  effort.  This  volume  provides  basic 
information  for  Che  T&E  planner  to  deal  with  the  ET  component.  Other 
volumes  in  this  series,  notably  Volianes  4,  5,  6,  and  8,  may  provide 
greater  insight  into  the  ET  development  process. 
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No  cookbook  Approach  to  planning  and  conducting  tests  and  evalua¬ 
tions  relevant  to  ET  can  be  offered.  The  variations  in  procurement 
practices,  end-item  systems,  training  systems,  and  ET  subsystem  designs 
are  far  too  great  to  permit  a  simplistic  approach.  What  is  provided  is 
an  ET-oriented  approach  to  T&E  which  can  add  to  the  effectiveness  of  a 
T&E  program. 

As  indicated  in  Figures  1  and  2,  this  guidance  is  specifically 
related  to  the  principal  stages  of  the  acquisition  process  prior  to 
Milestone  III.  Because  of  the  variability  in  development  schedules,  it 
is  not  useful  to  deal  with  the  acquisition  process  in  greater  detail. 

This  document  is  divided  into  three  major  sections,  plus  three 
Appendixes.  This  section  provides  a  discussion  of  the  goals  appropri¬ 
ate  to  T&E  of  ET  designs  as  they  evolve  and  a  brief  discussion  of  the 
advantages  of  the  concept  of  Continuous  Evaluation  (CE). 

The  second  section  of  this  document  outlines  the  content  and 
format  requirements  for  preparing  and  updating  ET  inputs  to  a  Test  and 
Evaluation  Master  Plan  (TEMP).  The  guidance  included  in  this  section 
is  applicable  to  different  systems  and  to  TEMPs  prepared  at  different 
stages  of  system  development. 

The  third  section  discusses  three  issues  that  distinguish  ET 
testing  from  the  traditional  testing  appropriate  for  all  developing 
systems.  Approaches  for  addressing  these  issues  are  presented. 
Typically,  training  is  not  tested  as  part  of  the  prime  system 
development  process.  However,  ET  is  an  integral  part  of  the  end-item 
configuration.  This  means  that  ET  must  be  tested  as  it  is  developed, 
since  with  ET  the  degrees  of  freedom  to  add  to  or  modify  the  nature  of 
the  training  decrease  rapidly  as  the  design  of  the  overall  system  is 
performed . 

Appendix  A  consists  of  a  reproduction  of  tne  material  in  AMC 
Pamphlet  70-2,  Materiel  Acquisition  Handbook,  Chapter  13,  Test  and 
Evaluation.  This  material  describes  T&E  procedures  in  terms  of  the 
Army  Streamlined  Acquisition  Process  (ASAP). 

Appendix  B  discusses  two  issues  which,  while  not  a  part  of 
traditional  T&F,  impact  the  training  system  development  process  and  its 
quality  control  program.  These  issues  are:  (1)  need  for  early  and 
continuous  involvement  of  training  developers  with  ET  responsibility  as 
essential  members  of  the  system  design  team;  and  (2)  need  for  an 
oversight  process,  in  addition  to  conventional  T&E,  that  assures  that 
the  processes  needed  to  define,  design,  and  implement  effective  ET  are 
actually  conducted  during  system  development. 

Appendix  C  provides  a  list  of  abbreviations  and  acronyms. 
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Requirements  Proof  of  Development  Production  and 

and  Tech.  Base  Principle  Production  Prove  Out  Deployment 


Figure  2.  The  Army  Streamlined  Acquisition  Process  and  associated  test  and 
evaluation  events. 


Goals  of  ET  Testing 


The  ultimate  goal  of  ET  T&E  is  to  assure  effective  soldier 
performance  under  combat  conditions  in  the  field.  T&E  subgoals  for  ET 
match  the  testing  goals  stated  in  Army  Regulation  (AR)  70-10,  Test  and 
Evaluation  During  Development  and  Acquisition  of  Materiel.  These  goals 
are: 


1.  Demonstrate  how  well  a  materiel  system  meets  its  technical 
and  operational  requirements. 

2.  Provide  data  to  assess  developmental  and  operational  risk 
for  decision  making. 

3.  Verify  that  technical,  operational,  and  support  problems 
identified  in  previous  testing  have  been  corrected. 

4.  Ensure  that  all  critical  issues  which  could  be  resolved  by 
testing  have  been  adequately  considered. 

AR  70-10,  along  with  AR  71-3,  User  Testing ,  defines  the  require¬ 
ments  for  system  testing  during  acquisition.  This  test  and  evaluation 
process  is  illustrated  in  Figures  1  and  2.  These  figures,  adapted  from 
an  older  version  of  DARCOM  Pamphlet  70-2,  Materiel  Acquisition  Hand¬ 
book,  portray  the  T&E  process  as  it  occurs  in  the  ASAP,  as  well  as  in 
the  traditional  Life  Cycle  Systems  Management  Model  (LCSMM).  The 
discussion  of  relevant  T&E  issues  for  ET  is  referenced  to  the  processes 
and  events  shown  in  Figures  1  and  2.  Figures  1  and  2  also  relate  ASAP 
ET  T&E  activities  to  those  appropriate  within  the  LCSMM. 

This  document  is  designed  for  both  Government  and  contractor 
personnel  responsible  for  the  specification  of  test  issues  and  the 
conduct  of  tests.  The  contractor  will  be  responsible  for  much  of  the 
training  effectiveness  evaluation  as  the  ET  design  evolves  and  as 
specific  courseware  is  generated.  Traditionally,  both  Government  and 
contractor  TSE  activities  emphasize  the  verification  of  equipment 
operation  in  accordance  with  design  specifications.  This  will  continue 
to  be  true  for  systems  involving  ET.  Contractor  ET  T&E,  in  addition, 
should  particularly  concentrate  on  verification  of  the  feasibility  of 
ET  design  approaches,  the  overall  system  compatibility  of  the  ET 
hardware  and  software,  and  the  effectiveness  of  specific  modules  of 
courseware . 

The  nature  of  the  ET  design  process  implies  that  if  ET  T&E  is 
rigorously  applied  during  development,  the  resulting  ET  subsystem  will 
be  operationally  suitable.  However,  if  problems  are  not  identified  and 
rectified  until  system  fielding,  the  degrees  of  freedom  for  achieving 
successful  "fixes"  will  be  reduced.  Therefore,  this  volume  is  particu¬ 
larly  concerned  with  ET  T&E  during  the  development  process. 

As  is  true  of  the  end-item  itself,  the  results  evaluated  by  ET  T&E 
are  greatly  impacted  by  the  scope  and  process  of  the  ongoing  analysis 
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and  design  efforts.  The  assessment  of  some  of  these  factors  is 
discussed  in  Appendix  B.  These  are  issues  which  are  critical  to  the 
successful  integration  of  ET  with  the  end-item  system.  Since  many  ET 
analyses  must  be  completed  prior  to  end-item  design,  this  may  make 
early  performance  assessment  (or  at  least  some  form  of  directed  over¬ 
sight)  appropriate,  to  assure  that  ET  design  is  effectively  dovetailed 
with  the  prime  system  design,  even  though  this  is  not  \dien  training  is 
usually  assessed. 


The  Continuous  Evaluation  (CE)  Concept  and 
Activities  for  ET 


In  its  fully  embedded  form,  ET  requires  that  the  training  hardware 
and  software  be  an  integral  part  of  the  complete  operational  system, 
whether  it  is  a  combat  system  or  a  combat-support  system.  This  means 
that  the  design  of  the  ET  subsystem  must  b^  accomplished  at  the  same 
t ime  and  in  concert  with  the  design  of  the  prime  item  or  "mission" 
system.  Because  of  the  complexity  of  most  modern  systems  for  which  an 
ET  approach  should  be  considered,  simultaneous  development  is  necessary 
in  cases  even  when  the  ET  subsystem  will  be  adjunct  or  appended  rather 
than  built  in,  The  degree  of  interaction  necessary  between  training, 
operational  software,  and  databases,  and  the  fact  that  ET  is  presented 
to  the  soldier  at  his  work  station,  link  the  training  function  with  all 
other  system  operational  functions  for  design  purposes. 

The  fact  that  the  ET  development  must  be  continuously  integrated 
with  the  design  process  begs  the  question  of  testability  of  the 
product.  If  tho  T&E  effort  is  instituted  only  after  a  complete  train¬ 
ing  product  is  available,  and  the. product  is  found  wanting,  what  can  be 
done?  It  is  too  late  to  institute  major  changes.  Continuous  Evalua¬ 
tion,  a  new  concept  under  exploration  by  the  T&E  community,  may  provide 
the  answer. 

Continuous  Evaluation  (CE)  is  a  concept  designed  to  facilitate 
system  development  and  thorough  performance  evaluation.^  Unlike  the 
traditional  approach,  wherein  the  majority  of  systematic  evaluation 
occurs  during  scheduled  periods,  CE  offers  the  opportunity  for  much 
earlier  assessment  of  appropriate  ET  design.  CE  enables  all  portions 
of  the  de'^e loping  system  to  be  assessed  and  tuned  as  part  of  Che 
ongoing  design  process  Co  achieve  maximum  ef fecti^'eness.  This 
aojustment  can  be  applied  to  the  ET  subsystem  in  the  same  way  it  is 
applied  to  hardware  and  software  subsystems.  CE  is  increasingly 
valuable  as  a  function  of  system  complexity;  it  is  also  increasingly 
valuable  as  a  function  of  Che  complexity  and  sophistication  of  the 
system's  ET  component. 


^Kaplan,  J.  (1985).  Continuous  evaluation:  A  concept.  Journal  of 
Operational  Test  and  Evaluation .  VT ( k ) . 
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Early  on  in  the  evaluation  process  for  ET,  before  there  are  major 
assemblies  of  hardware  and  software,  system  functions  assigned  to  ET 
can  and  should  be  assessed  function-by-function  in  a  process  of  forma¬ 
tive  evaluation.  This  process  should  utilize  any  component  performance 
data  v^ich  exist,  and  logically  predict  performance  for  all  training 
functions.  As  design  proceeds,  this  process  is  iterated  at  succes¬ 
sively  greater  levels  of  system  and  ET  subsystem  integration  to  confirm 
that  planned  functions  are  feasible  and  cost  effective.  This  same 
component-by-component,  subassembly-by-subassembly,  subsystem-by¬ 
subsystem  evaluation  approach  can  continue  to  be  followed  as  function 
analysis  evolves  to  component  analysis.  Initially,  functions  are 
modeled  utilizing  data  derived  from  rational  analysis.  Later,  these 
"soft"  data  can  be  replaced  by  actual  test  results.  As  the  testing 


process  commences,  the  design  can  be  reconfigured  to  continually  keep 
the  differences  between  needed  performance  and  actual  ET  subsystem 
capability  as  small  and  manageable  as  possible. 

The  nature  of  appropriate  T&E  is  to  assure  effective  ET  design 
integration  and  functional  capability  changes  as  the  system  design 
evolves.  Six  types  of  T&E  activities  are  appropriate: 

1. 

Oversight  to  assure  that  training  system  developers  work  in 
concert  with  operational  system  designers. 

2. 

Oversight  to  assure  that  analyses  done  to  configure  and 
size  the  operational  system  and  the  ET  subsystem  are  being 
appropriately  accomplished. 

3. 

Evaluation  Co  confirm  that  the  ET  design  will  function  as 
needed  to  deliver  the  required  training. 

4. 

Evaluation  to  demonstrate  that  the  designed  prime  system 
capabilities  (hardware  and  software)  can  accommodate  Che 
anticipated  ET. 

5. 

Test  to  assure  that  the  developing  ET  courseware  will  teach 
when  and  where  needed  (user  test  [UT],  skill  acquisition, 
and  skill  sustainment). 

6. 

T&E  to  confirm  that  the  ET  component  evolves  successfully 
with  the  changing  operational  system. 

The  formal  TEMP  is  configured  to  deal  with  activities  3  through  6 
above.  Items  1  and  2  are  not  normally  in  the  purview  of  T&E,  and, 
while  extremely  important,  are  not  discussed  in  the  main  body  of  this 
report.  They  are  discussed  in  Appendix  B. 

The  TEMP  is  one  key  to  successful  ET  development.  As  indicated  in 
Figures  1  and  2,  the  TEMP,  in  its  successive  iterations,  structures  the 
overall  T&E  program  during  system  development.  Insightful  integration 
of  T&E  planning  for  ET  with  T&E  for  the  prime  system  can  assure  that 
the  EX  portion  of  the  trai.ning  system  will  function  to  match  human 
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performance  (individual  and  crew)  to  overall  system  performance, 
next  section  of  this  document  discusses  the  requirements  for,  and 
preparation  of,  the  ET  portions  of  a  TEMP. 


The 
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INTEGRATING  ET  INTO  THE  TEST  AND  EVALUATION  MASTER  PLAN  (TEMP) 


The  TEMP  is  "...the  basic  planning  document  for  all  T&E  related  to 
a  particular  system  acquisition..."^.  This  document  establishes  the 
requirements  and  timetable  for  T&E  during  the  acquisition  process.  It 
is  the  working  document  produced  for  T&E  guidance  by  the  Test  Integra-’ 
tion  Working  Group  (TIWG).  The  TEMP  documents  the  goals  of  T&E  inves¬ 
tigations,  the  T&E  schedule,  and  projected  resources. 


Foundation  of  the  TEMP 


The  TEMP  changes  during  the  acquisition  process.  If  the  LCSMM  is 
in  place,  then  there  are  three  iterations  of  the  TEMP;  one  at  Mile¬ 
stone  I,  to  plan  for  T&E  during  Demonstration  and  Validation;  one  at 
Milestone  II,  for  T&E  during  Full-Scale  Development;  and  one  at  Mile¬ 
stone  III,  for  T&E  during  Production  and  Deployment.  If  the  ASAP  Is  in 
place,  then  TEMPs  I  and  II  are  combined  and  developed  for  T&E  during 
Development  Production  Prove  Out,  while  TEMP  III  again  plans  for 
Production  and  Deployment. 


ET  Considerations  in  the  TEMP 


The  purpose  of  the  TEMP  is  to  identify  the  technical  and  opera¬ 
tional  issues  relevant  to  T&E.  Ideally,  each  test  should  be  identified 
and  its  goals  specified.  These  goals  are  stated  in  AMC/TRADOC  Pamphlet 
70-2,  Materiel  Acquisition  Handbook  (Appendix  A).  With  respect  to 
training,  the  goals  of  T&E  are  to  determine  that  training  requirements 
are  met,  and  to  establish  compatibility  and  interoperability  with  the 
Army  and  other  services  (AMC/TRADOC  Pam  70-2,  pg.  A-3,  point  d.). 
However,  these  goals  are  insufficient  with  respect  to  T&E  for  ET. 

By  the  time  the  development  and  prime  system  acquisition  process 
is  complete,  more  than  simply  the  ability  of  ET  to  meet  training 
objectives  must  be  established.  To  establish  the  proper  goals  for  ET 
T&E,  it  is  necessary  to  consider  goals  originally  intended  for  the 
operational  capability  of  the  prime  equipment.  These  additional  goals 
are  to  assess:  operational  suitability,  operational  effectiveness,  and 
readiness.  Applying  these  goals  to  ET,  it  is  necessary  to  address: 


^HQ,  Department  of  the  Army.  (1986).  Research,  Development,  and 
Acquisition:  Test  and  Evaluation  (AR  70-10,  30  April  1986) . 


1.  ET  suitability.  Suitability  means  tAiether  ET  is  an 
appropriate  training  method  for  this  system  and  its  use 
concept,  and  \^ether  the  correct  soldiers  are  targeted  to 
use  it. 

2.  ET  effectiveness.  Effectiveness  includes  adequacy  of 
content  coverage,  success  of  training  transfer  to  job 
performance,  and  related  issues. 

3.  The  ability  of  ET  to  contribute  to  soldier  readiness.  This 
implies  that  ET  must  target  and  affect  the  particular 
aspects  of  soldier  performance  that  must  be  ti.ainrd  or 
sustained  to  improve  force  readiness. 

This  change  in  goals  is  necessary  because  development  of  ET  does 
not  take  place  following  equipment  development  as  has  historically  been 
the  practice  in  traditional  trainii^  system  development.  Instead,  ET 
is  developed  in  concert  with  prime  equipment  development.  As  has  been 
discussed,  once  the  equipment  is  developed,  a  large  part  of  the  func¬ 
tionality  to  implement  ET  is  fixed.  If  any  needed  elements  to  imple¬ 
ment  ET  functions  are  omitted,  it  is  unlikely  that  they  can  be  added-in 
economically  later.  This  alteration  in  the  phasing  of  training  devel¬ 
opment  relative  to  ET  is  dealt  with  in  Volume  10  of  this  series  ( Inte¬ 
grating  ET  into  Acquisition  Documentation).  ET  must  also  be  tested  and 
evaluated  on  a  schedule  in  concert  with  the  development  of  the  prime 
system  hardware  itself. 


Integrating  TEMP  With  Continuous  Evaluation 

The  CE  approach  means  that  parts  of  a  materiel  system  are  evalu¬ 
ated  for  their  ability  to  achieve  required  goals  as  they  are  developed. 
This  concept,  recently  implemented  for  hardware  and  software,  should  be 
implemented  for  ET  as  well.  There  are  three  issues  that  pertain  to 
implementing  a  CE  approach  to  ET.  These  issues  must  be  reflected  in 
TEMP  decisions,  and  are  referred  to  later  in  this  section.  They  are: 

1.  ET  must  be  designed  in  concert  with  the  materiel  system 
design,  whether  ET  is  fully  integrated  or  is  an  adjunct 
system. 

2.  Technical  testing  of  ET  must  occur  while  there  is  still 
time  to  remedy  deficiencies. 

3.  ET  should  be  tested  piecemeal  (i.e.,  as  elements  of  the  ET 
capability  evolve  to  points  where  they  can  be  meaningfully 
tested;  testing  should  not  be  delayed  until  all  elements  of 
ET  are  fully  developed),  as  the  materiel  and  ET  designs 
progress . 
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EX  Goals 


As  the  first  step  in  developing  the  TEMP  for  ET,  the  training 
developers  (whether  or  not  members  of  the  TIWG)  should  clarify  the 
goals  of  ET  for  the  particular  system  under  development  to  the  TIWG. 
This  should  be  done  whether  ET  is  directed  from  higher  authority  or 
whether  it  is  one  of  a  number  of  training  methodologies  under  consider¬ 
ation  for  the  new  system.  This  clarification  will  help  the  TIWG  under¬ 
stand  the  importance  of  different  aspects  and  applications  of  ET,  and 
to  make  the  necessary  testing  decisions.  To  provide  knowledgeable 
resources  and  oversight  for  considering  the  testing  of  ET  and  other 
training  system  elements,  it  is  suggested  that  the  TIWG  include  voting 
representation  from  the  training  community.  This  might  include 
membership  from  the  DirectorateC s)  of  Training  and  Doctrine  (DOTD)  of 
the  proponent  school(s). 

The  following  questions  will  help  to  establish  the  ultimate 
performance  expectations  for  the  ET  component.  These  questions  probe 
key  points  to  be  resolved  prior  to  specific  T&E  planning: 

1.  Is  ET  intended  for  acquisition  training? 

a.  If  the  answer  is  yes,  i»ill  ET  be  used  at  institu¬ 
tional  sites,  unit  sites,  or  both? 

b.  Which  training  capabilities  will  be  used  at  each 
site? 

The  content  of  acquisition  training  is  often  quite  differ¬ 
ent  from  the  content  of  sustainment  training.  The  former 
is  heavily  oriented  toward  the  initial  acquisition  of 
skills  at  the  entry  job  performer  level  ;  sustainment  is 
usually  concerned  with  skill  integration  in  the  operational 
setting,  and  the  refinement  of  already  acquired  skills. 
Sustainment  and  related  training  roles  (e.g.,  skill  level 
upgrade,  train-up  of  replacement  personnel  in  mobilization 
situations)  may  be  associated  with  needs  for  higher  fidel¬ 
ity  levels  and  larger  varieties  of  training  scenarios  when 
simulation  is  involved. 

2.  Is  ET  intended  for  sustainment  training  at  unit  sites? 

a.  If  ET  is  intended  for  unit  sustainment  training, 

will  training  be  individualized  for  particular  sites 
and  applications  (e.g.,  scenarios  customized  for  the 
location  of  deployment)? 

If  training  is  intended  for  unit  sites,  then  it  will  have 
broader  application  if  it  does  not  rely  on  instructors  as 
active,  real-time  performance  monitors.  Training  designed 
for  institutional  application  can  rely  to  a  greater  extent 
on  instructors  to  perform  evaluations  and  deliver 
feedback. 
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3.  Are  there  identifiable  sets  of  training  objectives? 

a.  For  different  types  of  personnel  (e.g.,  operators 
versus  tnaintainers}? 

b.  For  different  levels  of  training  (e.g.,  gunner 
versus  crew  chief,  MOS  XXXIO  versus  MOS  XXX50)? 

The  types  of  trainees  will  have  a  strong  bearing  on  the 
types  of  T&E  that  are  developed.  At  various  points  in 
development,  ET  must  be  evaluated  for  each  type  of  trainee, 
since  the  types  of  tasks  trained,  and  even  the  hardware 
used  in  training,  may  differ  for  different  job  categories. 
For  example,  a  chief  of  section  may  utilize  ET  that 
presents  battle  scenarios  on  a  display,  vdiile  a  maintainer 
may  utilize  ET  built  into  a  separate  diagnostic  device  that 
attaches  to  the  actual  equipment. 

4.  Is  ET  intended  for  soldier  and  unit  preparedness  evaluation 
(e.g.,  ET  measurement  capabilities  used  as  part  of 
evaluations  in  the  ARTEP  process)? 

If  soldier  and  unit  preparedness  evaluation  are  intended, 
then  T&E  must  en.sure  that  provision  is  made  for  features 
necessary  to  achieve  valid  evaluations.  The  following 
features  are  important  to  all  training,  but  are  especially 
critical  to  evaluation: 

a.  Data  collection  and  recording  capabilities  for 
performance  assessment  must  be  able  to  deal  with 
large  amounts  of  data. 

b.  Performance  data  must  be  kept  secure  from  partici¬ 
pants  and  secure  from  tampering. 

c.  A  range  of  scenarios  varying  in  difficulty  must  be 
available,  so  that  the  full  range  of  individual 
and  unit  performance  can  be  measured . 

d.  A  variety  of  scenarios  must  be  available  at  each 
level  of  difficulty,  so  that  individuals  and  units 

^cannot  familiarize  themselves  with  the  test 
scenarios  and  raise  their  scores  solely  due  to  this 
familiarity. 

The  first  two  of  these  features  may  be  evaluated  during 
technical  tests  of  the  ET  subsystem.  The  last  two  features 
may  be  evaluated  during  First  Article  Testing  (FAT)  during 
Production  and  Deployment. 
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Content  of  the  TEMP 


Now  that  ET  T&E  has  been  put  in  perspective  with  the  TEMP  and 
related  evaluation  processes,  the  specifics  of  the  TEMP  with  regard  to 
ET  are  presented.  The  TEMP  is  a  dynamic  document,  changing  throughout 
the  course  of  the  system  development  and  procurement  process(es).  In 
addition,  different  procurements  will  have  more  or  less  specific  TEMPs, 
and  the  particular  points  that  the  TEMP  deals  with  will  differ  across 
procurements.  The  following  discussion  presents  an  ideal  situation  in 
which  ET  is  a  significant  aspect  of  the  procurement.  Accordingly,  the 
discussion  notes  many  points  where  ET  should  be  dealt  with.  It  is  the 
option  of  the  TIWG  to  decide  how  much  of  this  content  should  be 
included  in  a  given  TEMP. 

Specific  guidance  on  the  format  of  a  TEMP  is  provided  in  DOD 
Directive  5000. 3M-1,  Test  and  Evaluation.  The  TEMP  is  divided  into 
five  parts  (shown  in  Roman  numerals): 


Part  I.  System  Details  and  Description. 

Part  II.  Program  Summary. 

Part  III.  Technical  Test  and  Evaluation  (TT&E)  Outline. 

Part  IV.  User  Test  and  Evaluation  (UT&E)  Outline. 

Part  V.  T&E  Resources  Summary. 

Administrative  information,  which  comes  first,  is  unchanged  by  ET. 
There  are  also  some  schedules  that  are  appendixes  to  the  TEMP.  These 
are  also  unchanged  by  ET.  The  TEMP  paragraphs  to  be  discussed  in  terms 


of  ET  are: 

1. 

System  Description 

(Part 

I) 

2. 

Required  Technical  Characteristics 

(Part 

I) 

3. 

Management  Summary 

(Part 

II) 

4. 

Integrated  Schedule 

(Part 

II) 

5. 

Critical  Technical  Issues 

(Part 

III) 

6. 

TT&E  Events 

(Part 

IV) 

7. 

UT&E  Events 

(Part 

IV) 

8. 

Test  Resources  Summary 

(Part 

V) 
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System  Description 


The  system  description  should  include  a  statement  of  the  fact  that 
ET  is  part  of  the  system.  For  TEMP  I  in  the  LCSMM,  this  may  be  a 
statement  that  ET  is  under  consideration.  Interfaces  should  be 
described  for  adjunct  or  strap-on  ET,  and  for  any  planned  netted  opera¬ 
tions  (i.e.,  interconnecting  the  ET  components  of  two  or  more  systems 
to  enable  collective  training  above  the  operator  or  crew  level)  to  be 
considered  for  inclusion  in  the  ET  concept. 

ET  should  be  considered  a  unique  characteristic  for  Army  equipment 
at  the  present  time.  This  determination  must  be  made  in  historical 
context,  however.  Once  ET  becomes  commonplace,  it  may  no  longer  be 
appropriate  to  call  it  unique.  However,  the  procedures  required  for 
integrating  and  testing  ET  are  new  enough  that  it  is  appropriate  to 
denote  ET  as  a  unique  feature. 


Required  Technical  Characteristics 


ET  should  be  named  as  a  required  technical  characteristic  of  the 
prime  system.  Some  specific  ET  technical  characteristics  are  specified 
in  the  following  list.  The  characteristics  may  be  presented  in  tabular 
form: 

1.  Allowable  times  to  transition  into  and  out  of  ET. 

2.  Allowable  logistics  burden  for  ET  and  any  appended 
components.  Appended  components  allow  ET  when  certain 
support  components  (e.g.,  scenario  generator)  are  added  to 
the  system  temporarily. 

3.  Levels  or  types  of  training  to  be  delivered  by  ET  (e.g., 
acquisition,  sustainment,  expert). 

4.  Number  of  scenarios  and  the  required  variability  of 
training  (e.g.,  location-specific  scenarios,  expert-level 
scenarios).  Also,  sufficient  variety  to  prevent 
memorization  and  boredom. 

5.  Training  content  or  topics  (e.g,,  failure  modes,  multiple 
targets,  maintenance). 

6.  Allowable  extent  of  dependence  on  instructors  or  training 
managers  during  ET  sessions. 

7.  Required  level  of  fidelity  of  simulations.  Each  aspect 
should  be  dealt  with  in  as  much  detail  as  possible  (e.g., 
visual  scene  fidelity,  representation  fidelity  of  communi¬ 
cations;  dummy  rounds).  For  TEMP  1,  this  specification 
will  probably  be  functional  rather  than  technical  (e.g., 
visr.al  fidelity  good  enough  to  allow  detection  [or  identi¬ 
fication,  as  appropriate  to  the  scenario]  of  a  helicopter 
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at  the  appropriate  distance  for  missile  launch),  but 
technical  specifications  are  preferred.  The  fidelity 
parameters  for  each  aspect  of  simulation  should  be  stated 
clearly  enough  to  allow  piecemeal  evaluation.  For  example, 
visual  simulations  should  specify  display  resolution, 
movement ,  verisimilitude,  etc.  This  way,  the  suitability 
of  software,  display  generators,  monitors,  etc.  can  be 
determined . 

8.  Performance  evaluation  and  assessment  capability. 

9.  Performance  data  security. 

10.  Feedback  in  support  of  effective  learning. 

11.  Adaptability,  so  Chat  Che  system  can  respond  to  soldier 
performance  and  adapt  the  presented  training  materials  to 
strengthen  weak  areas. 


Management  Summary 

This  is  the  first  section  of  Part  II,  Program  Summary.  This 
section  should  specify  the  agency(ies)  responsible  for  ET  T&E. 


Integrated  Schedule 

The  integrated  schedule  is  a  critical  part  of  the  ET  TiiE  effort, 
because  ET  T&E  must  take  place  concurrent  with  system  T&E.  Without 
this  coordination,  it  will  be  difficult  or  impossible  to  implement 
changes  to  ensure  satisfactory  ET.  In  the  past,  usual  practice  has 
been  that  training  products  have  been  procured  separately,  and,  hence, 
have  had  separate  TIWGs  that  include  training-knowledgeable  members. 

The  normal  time  for  developing  and  testing  training  has  traditionally 
been  after  the  system  concepts,  hardware,  aixl  software  are  firm.  At 
the  present  time,  the  non-training-oriented  members  of  the  TIWG  may  not 
be  aware  of  the  requirement  to  address  ET  and  integrate  it  into  the 
testing  issues  and  schedule.  Training  representatives  must  ensure  that 
the  proper  elements  are  included  in  the  schedule  at  the  proper  times. 

The  ET-related  part  of  the  integrated  T&E  schedule  should  take 
note  of  scheduled  hardware,  software,  and  training  delivery  dates,  and 
hardware  and  software  test  dates.  These  should  be  used  to  forge  a 
continuous  evaluation  program  for  ET,  in  which  each  component  related 
to  ET  is  evaluated  for  its  ability  to  fulfill  its  ET  functions  at  the 
same  time  it  is  evaluated  for  its  ability  to  fulfill  its  operational 
functions  or  mission.  This  relationship  is  reflected  below  in  TEMP  I, 
Activity  2,  and  TEMP  II,  Activity  1. 
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TEMP  I  should  plan  for  the  following  activities: 

1.  Establish  overall  goals  of  ET  (training  audience, 
collective  and  individual  tasks  to  be  trained,  training 
sites,  introductory  vs.  sustainment,  level  of  proficiency 
to  be  achieved).  Perform  PTEA. 

2.  Establish  what  equipment  (real  or  models)  will  or  can  be 
available  with  ET  capability  that  can  be  evaluated,  and  on 
what  schedule.  The  d«>'lvery  schedule  for  ET-related 
components  may  dictate  the  order  in  fdiich  piecemeal 
evaluation  for  ET  takes  place.  To  ensure  that  deficiencies 
can  be  corrected,  each  relevant  piece  of  hardware  and 
software  should  be  evaluated  against  its  ET>oriented 
functional  requirements.  Simulation-related  items  must 
have  appropriate  fidelity;  interfaces  must  allow  LT  access 
to  required  hardware  and  software  for  simulation  and 
performance  assessment;  hardware  must  have  provision  for 
varied  ET  scenarios;  soldier-machine  interfaces  must 
provide  for  rapid  transitioning  into  and  out  of  ET ;  and  so 
forth . 

3.  Determine  v^ether  ET  will  support  user  test  (UT)  of  the 
prime  system  (not  just  ET  evaluation). 

A.  Establish  that  personnel  will  be  available  as  test  subjects 
for  ET  performance  evaluation  during  UT. 

5.  Arrange  for  subject-matter  experts  (SMEs)  to  assist  in 
formative  evaluations  of  training  system  components  in 
concert  with  CE.  SMEs  may  also  be  useful  as  overseers  of 
summative  performance  evaluations  of  ET  (in  later  test 
events). 

6.  Evaluate  ET  training  objectives  to  ensure  complete  coverage 
of  all  areas  requiring  ET. 

7.  Verify  feasibility  of  ET  approach,  based  on  prime  system 
hardware  configuration.  This  is  the  essence  of  the 
verification  that  ET  can  be  "embedded." 

Items  1,  2,  3,  and  7  are  aimed  at  ensuring  ET  suitability.  Items 
A,  5,  and  6  are  aimed  at  ensuring  training  effectiveness.  Item  A  will 
also  contribute  to  evaluating  ET's  contribution  to  soldier  readiness. 

TEMP  II  should  plan  for  the  revision  and  completion  of  the  above 
activities,  especially  point  6.  In  addition,  the  following  activities 
should  be  planned  (the  order  reflects  the  approximate  sequence  in  idiich 
the  actual  activities  will  take  place): 

1.  Schedule  ongoing  developmental  review  of  the  ET  courseware 
material.  Periodic  reviews  should  cover  pieces  of  ET  as 
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they  are  produced.  Hardware  capabilities  are  an  important 
subject  of  this  review  process.  If  possible,  schedule 
exercises  \diich  demonstrate  how  ET  will  utilize  the 
hardware  in  an  actual  training  context.  For  instance,  it 
may  be  possible  to  select  one  scenario  for  one  lesson  and 
prepare  that  scenario  completely--cext ,  visual,  response 
processing,  feedback,  etc.  This  scenario  would  then  be 
presented  to  SMEs,  trainers,  and  perhaps  users  to  evaluate 
training  approach,  fidelity,  and  adequacy  of  training 
implementation.  These  reviews  may  take  place  in-plant. 

2.  Establish  v^ich  aspects  of  the  ET  component  can  be  tested 
with  performance  evaluation,  given  the  expected  level  of  ET 
and  system  development. 

3.  Specify  experimental  design  of  performance  (user)  tests. 

4.  Schedule  subjects  for  in-plant  performance  evaluation  of 
the  ET  component. 

5.  Schedule  equipment  for  performance  evaluation  of  ET 
component . 

6.  Evaluation  of  ET  courseware  outlines  or  lesson  plans  for 
adequacy. 

7.  SHE  review  of  ET  training  material. 

8.  Prior  training  for  test  subjects  for  the  performance 
evaluation  of  sustainment  ET. 

9.  ET  performance  (user)  evaluation. 

All  items  will  help  to  establish  ET  effectiveness.  Items  2,  3,  4, 
5,  8,  and  9  are  also  aimed  at  evaluating  ET's  contribution  to  soldier 
read iness  . 


Critical  Technical  Issues 

This  is  the  first  section  of  Fart  III  of  the  TEMP,  the  T&E 
outline.  This  section  is  the  Technical  T&E  (TT&E)  Outline.  Training 
issues  may  be  subsumed  under  a  subheading  called  Training. 

The  issues  noted  above  as  required  technical  characteristics 
should  be  scheduled  for  test  in  this  section.  Some  tests  can  be 
accomplished  as  part  of  an  in-plant  test  of  the  ET  component,  using  a 
physical  model  of  the  actual  system  that  contains  the  ET  component. 
However,  there  is  a  high  probability  that  ET  courseware  will  not  be 
completed  by  the  early  test  dates.  The  TIWC  should  direct  the 
contractor  to  develop  FT  courseware  necessary  for  the  in-plant  test 
such  that  the  critical  elements  of  the  prime  system  and  its  ET 
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component  can  be  evaluated.  This  approach  is  in  keeping  with  the  CE 
concept  of  piecemeal  testing  to  ensure  timely  recommendations  for 
revision  to  remedy  def ic ievicies  . 

As  part  of  its  input  to  the  procurement  contract,  the  TIWG  should 
ensure  that  there  will  be  testable  taaterial  to  allow  evaluation  of  each 
critical  part  of  the  ET  component.  Attention  should  be  paid  to  the 
following  issues: 

1.  Quality  of  required  simulation.  Is  the  representation  and 
content  of  visual  material  satisfactory? 

2.  Has  provision  been  made  for  netted  operations?  It  may  be 
impossible  to  test  the  achievement  of  netted  operations 
until  a  quantity  of  testbed  articles  or  even  operational 
system  prototypes  have  been  produced. 

3.  Is  the  human  interface  for  ET  satisfactory  in  all  respects 
for  all  ET  functions,  from  ET  initiation,  through  training 
operations,  to  return  to  operational  mode?  Has  the 
interface  been  designed  to  assure  no  negative  transfer  and 
no  confusion  regarding  idiich  mode  the  system  is  in? 

4.  Does  trainee  performance  assessment  capability  exist  and 
has  provision  been  made  for  instructors  or  monitor  student 
performance? 

5.  Is  the  general  nature  of  training  feedback  during  ET 
acceptable  (frequency,  immediateness,  completeness)? 

6.  Is  the  required  range  of  training  provided  (acquisition, 
sustainment,  etc.)? 

7.  Has  provision  been  made  to  vary  training  content? 

8.  Does  the  system  provide  safe  transition  between  training 
and  operational  modes? 

TT&E  Events .  In  some  TEMPs,  there  are  specific  sections  dealing 
with  particular  test  events.  ET  T&E  planners  should  make  every  effort 
to  ensure  that  specific  ET  tests  and  evaluations  are  included  in  the 
appropriate  test  events.  This  is  very  important  because,  as  has  been 
noted,  ET  is  not  yet  second  nature  to  the  various  members  of  the  TIWG 
or  to  system  managers.  One  critical  event  is  an  early  look  at  training 
capabilities  vdien  the  system  breadboard  is  reviewed  for  other  technical 
issues. 


Critical  Operational  Issues 

This  is  the  first  section  of  Tart  IV,  User  Test  and  Evaluation 
(UT&E).  The  operational  issues  will  be  tested  in  performance  (user) 
tests,  involving  actual  troops. 
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This  test  should  be  treated  as  if  it  were  a  long  lead-time  item, 
because  the  evaluation  requires  suitable  Army  personnel  as  test 
subjects.  Specifically,  a  test  of  ET  intended  for  initial  skills 
acquisition  training  should  include  at  least  some  soldiers  who  are 
representative  of  the  population  to  be  trained,  in  terms  of  aptitude, 
prior  training,  and  unfamiliarity  with  the  new  system  or  related 
systems.  The  government  agency  that  will  provide  these  personnel  must 
also  be  appraised  of  the  location  of  the  tests,  so  that  suitable  travel 
orders  and  arrangements  may  be  made.  Ordinarily,  the  earliest 
opportunity  for  performance  tests  will  be  at  the  contractor's  plant 
(in-plant  test),  necessitating  TDY.  The  TEMP  should  also  specify  the 
schedule  for  test  personnel  involvement.  Often,  only  limited  numbers 
of  test  subjects  can  be  trained  at  one  time  using  ET  on  the  test 
system. 

Another  issue  to  be  considered  during  development  of  the  TEMP  is 
whether  it  is  useful  and  feasible  to  contrast  ET  with  alternative 
training  methodologies.  That  is,  should  another  methodology  be  used  to 
train  soldiers  and  the  results  contrasted  with  ET  in  terms  of;  trans¬ 
fer  of  training,  time  to  train,  or  cost  to  train?  This  type  of 
contrast  is  not  usually  performed,  but  the  capability  may  be  inherent 
in  the  test  situation  if  some  are  to  be  trained  using  other  training 
methods,  perhaps  for  FOTE  or  Early  User  Test  and  Experimentation 
(EUT&E),  while  others  are  to  be  trained  using  ET  for  ET  evaluation.  A 
contrast  should  be  considered  if  the  data  appear  to  be  comparable,  to 
help  in  the  ultimate  decision  about  how  to  implement  ET. 

Test  Limitations.  This  topic  is  sometimes  included  as. a  sub¬ 
subsection  in  the  TEMP  under  UT&E.  An  important  issue  for  performance 
testing  is  the  availability  of  suitable  equipment  for  T&E  purposes. 

This  issue  is  dealt  with  elsewhere  in  this  document.  TEMP  1  should 
specify  the  type  of  equipment  and  capabilities  required  for  performance 
testing  of  ET.  Schedules  and  duration  of  tests  must  also  be  specified, 
so  that  ET  testing  can  be  coordinated  with  other  test  requirements. 

When  breadboard,  brassboard,  or  special  test  mockups  are  to  be 
used  for  EUT&E  of  ET,  they  must  first  be  evaluated  for  suitability. 
These  evaluations  may  be  part  of  TT&E.  The  particular  issues  to  be 
considered  are  the  environmental  fidelity  of  the  ET  testbed  (i.e.,  the 
similarity  of  the  ET  testbed  to  the  intended  ultimate  system  environ¬ 
ment),  and  the  extent  to  which  the  current  quality  of  ET  represents  the 
ultimate  quality  that  will  be  part  of  the  delivered  system. 


UT&E  Events 


If  there  is  to  be  an  in-plant  test,  possibly  using  a  brassboard  or 
other  special  testbed  for  performance  testing,  then  this  should  be 
specified  as  a  system-operational  issue  in  the  TEMP.  If  the  testbed  is 
not  dedicated  to  ET,  then  adequate  time  must  be  scheduled  on  the  exist¬ 
ing  testbed  for  ET  performance  testing. 
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Experience  has  shown  that  the  amount  of  time  required  for  ET 
performance  testing  is  often  underestimated.  First,  the  testers  must 
receive  training  on  how  to  use  Che  system  before  they  can  evaluate  how 
well  it  trains  user  subjects.  This  training  may  be  part  of  a  regular 
schedule  of  trainins  for  other  purposes,  but  it  must  be  conducted  early 
enough  for  the  performance  test  group  Co  plan  its  evaluations.  Second, 
Che  test  planners  must  assess  how  long  the  training  will  take,  and  how 
many  users  can  receive  it  during  any  given  time  period.  Third,  the 
test  planners  must  establish  how  much  training  a  user  can  be  expected 
to  absorb  in  a  given  period  of  time.  In  actual  practice,  ET  is  likely 
to  be  spread  over  a  long  time  period,  but  for  user  testing  this  time 
period  must  be  compressed.  Often,  the  limited  availability  of  training 
or  test  time  will  necessitate  separate  T&E  or  using  different  test 
subjects  (personnel)  for  different  training  functions  or  operational 
modes  . 


Test  Resources  Summary 

This  section  of  the  TEMP  is  likely  to  be  fairly  substantive  for  ET 
T&E,  because  a  number  of  special  resources  are  required  for  these 
activities.  More  important,  the  contractor,  and  perhaps  even  some 
government  personnel,  may  not  see  these  resources  as  directly  relevant 
to  the  procurement  of  the  prime  item  system.  Their  r  ’.al  relevance  is 
to  ensure  that  the  prime  item  system  will  be  used  properly  by  soldiers. 
The  specification  of  a  resource  presupposes  adequate  time  and  access  to 
that  resource  (not  always  the  case  unless  so  stated).  These  re-sources 
are ; 


1.  ET  demonstration  capability  at  the  breadboard  or  b^assboard 
stage  of  development  (Demonstration  and  Validation  or  Early 
Development  Production  Prove  Out). 

2.  Front-end  training  analysis  products  delivered  soon  enough 
to  perform  formative  evaluations  on  them  (e.g.,  training 
objectives,  courseware  outlines). 

3.  Piecemeal  access  to  hardware,  software,  and  training 
material  directly  related  to  ET  implementation  as  needed 
for  continuous  evaluation. 

4.  SMEs  for  evaluation  of  courseware  and  objectives, 
evaluation  of  testbed  adequacy,  and  for  evaluation  of 
physical  suitability  of  the  final  ET  conf  igurat  ii  n. 

5.  ET  testbed  during  late  Development  Production  Prove  Out  or 
Demonstration  and  Validation,  and  Full-Scale  Development. 

6.  Suitable  subjects  for  performance  (use^)  testing  (e.g., 
in-plant  testing).  Special  attention  must  be  paid  to  the 
time  period  ove  t^ich  subjects  required  for  evaluation  of 
sustainment  training  may  be  needed  (they  may  have  to  be 
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trained  at  one  or  more  points  in  time  and  retrained  and 
retested  one  or  more  months  later). 

7.  Suitable  preliminary  training  for  performance  test 
subjects,  especially  for  sustainment  testing.  It  may  be 
desirable  to  provide  comparison  training  using  alternatives 
to  £T  to  determine  whether  ET  affords  a  benefit. 

8.  Data  collection  hardware  and  software  included  in  the  test 
bed  to  allow  automatic  data  collection  during  the 
performance  (user)  test. 
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EMBEDDED  TRAINING  TEST  AND  EVALUATION  ISSUES 
AND  APPROACHES 


There  are  three  major  T&E  issues  derived  from  the  TEMP  and  related 
directly  to  the  T&E  applications  cited  earlier,  in  addition  to  planned 
general  critical  issues  (which  can  be  extracted  from  DOD  Directive 
5000.3,  Test  and  Evaluation).  Issue  1  below  involves  T&E  applications  3 
and  4  on  page  7,  while  issues  2  and  3  relate  to  applications  5  and  6. 
These  three  issues  are; 

1.  The  impact  of  ET  design  on  the  prime  system  design. 

2.  The  problem  of  T&E  for  sustainment  training. 

3.  The  requirements  for  ET  performance  testing. 

Resolution  of  these  issues  is  neither  standardized  nor  simple. 

What  is  an  appropriate  approach  for  a  particular  issue  on  one  system  is 
often  not  appropriate  for  the  same  iss'je  as  it  arises  in  the  develop¬ 
ment  of  another  system.  However,  all  of  these  issues  and  approaches 
are  important  and  require  attention.  To  the  degree  that  they  are 
ignored,  system  testing  will  be  deficient,  and  system  performance  can 
be  degraded. 

Existing  regulations  and  specifications  do  not  mandate  specific 
resolution  of  these  issues.  The  objective  of  this  discussion  is  to 
bring  the  issues  to  the  attention  of  system  development  and  evaluation 
personnel,  so  that  effective  tradeoffs  and  T&E  decisions  can  be  made. 


Issue  Number  1:  Impact  of  Embedded  Training  Design 
on  the  Prime  System 


Effective  integration  of  ET,  whether  it  is  appended  onto  or  built 
into  an  operational  system  design,  will  impose  changes  to  the  total 
operational  system;  changes  tdiich  would  not  occur  if  ET  were  absent 
from  the  overall  system  design  concept.  The  design  aim  is  to  maximize 
the  training  potential  of  the  ET  component  with  minimum  effect  on  the 
operational  system.  System  operational  capability  cannot  be  compro¬ 
mised.  The  system,  whether  a  weapon  system  or  a  support  system,  must 
meet  its  mission  requirements.  However,  there  are  broad  areas  of 
feasible  tradeoff  and  compromise  in  ^ich  characteristics  and  capabili¬ 
ties  of  the  total  operational  system  can  be  adjusted  to  maximize  both 
operations  and  training.  Generally,  the  tradeoffs  <diich  must  be 
accepted  to  achieve  effective  ET  include  those  listed  below. 
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Design  Tradeoffs 

Maintenance  load.  In  most  cases,  using  the  operational  system  for 
training  will  result  in  higher  use  factors  and  increased  hours  of 
operation.  This,  in  turn,  can  lead  to  a  greater  number  of  failures  and 
to  an  increased  maintenance  load  to  counter  them.  However,  experience 
indicates  that  maintenance  loads  can  be  minimized  by  concentrating  the 
high-use  areas  on  system  components  which  are  mainly  electrical  or 
electronic.  Actual  usage  data  indicate  that  only  when  increased  use 
involves  mechanical  "high  wear"  hardware  movements  (e.g.,  rotating  a 
tank  turret)  is  there  potential  for  unacceptable  maintenance  load.  The 
T&E  program  must  verify  that  ET  integration  does  not  compromise 
operational  reliability. 

Logistics  support  requirements.  The  addition  of  an  ET  subsystem 
will  impose  additional  maintainer  requirements.  By  and  large,  ET 
subsystems  involve  the  same  kinds  of  equipment  and  components  ^ich 
make  up  the  major  part  of  the  prime  operating  system.  Therefore,  the 
required  maintenance  skills  are  hardly  different  from  those  required  to 
maintain  the  operational  equipment.  Additional  maintenance  hours 
typically  will  be  required  to  counter  the  possibly  increased  number  of 
failures,  even  though  the  increase  may  be  small  in  most  cases. 

A  second  logistics  support  requirement  will  be  for  additional 
spare  parts  to  support  the  ET  subsystem,  as  well  as  additional  parts  to 
counter  the  possible  greater  number  of  failures  of  the  prime  item 
system.  It  may  also  be  necessary  to  make  some  portions  of  the  prime 
item  system  more  robust  than  would  be  necessary  if  the  system  were  not 
to  be  used  for  ET.  Increases  in  component  strength  and  durability  to 
counteract  the  increased  usage  imposed  by  training  may  be  required. 

T&E  activities  will  need  to  confirm  that  ET  logistics  decisions 
have  identified  and  allowed  for  all  significant  logistical 
implications. 

Cost  and  Training  Effectiveness  Analysis  (CTEA).  An  end  item 
system  with  an  ET  subsysteo  may  have  a  higher  acquisition  cost  and, 
perhaps,  a  higher  operatic  cost  than  it  would  have  without  the  ET 
subsystem.  However,  caref  '.alyses  which  sharply  define  the  role  of 
ET  in  the  total  training  systeu,  and  ET  designs  that  fulfill  the 
defined  role(s),  may  show  training  systems  including  ET  are  more 
cost-effective  \dien  compared  to  alternative  total  training  system 
configurations.  ET  will,  in  most  cases,  be  a  marginal  cost  shared  with 
other  software  development  activities.  As  such,  the  specific 
additional  cost  of  ET  may  be  very  low. 


Design  Impact  Questions 

Once  ET  has  been  splected,  after  considering  these  tradeoffs, 
three  T&E  questions  relating  to  the  impact  of  ET  on  operational  system 
design  will  also  require  attention: 
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1.  Is  it  more  cost  effective  to  have  the  ET  system  built  into 
the  operating  system  than  to  install  it  for  training  and 
then  dismount  and  store  it  between  training  uses?  Volume 

5  of  this  series  provides  guidance  to  the  process  of  making 
this  determination. 

2.  Has  sufficient  computer  capacity,  over  and  ^ove  that 
required  to  meet  initial  operational  mission  requirements, 
been  provided  to  permit  the  storage,  processing,  and 
display  of  intended  ET  courseware  ai^  simulation  data? 

This  particular  question  is  uniquely  significant  since 
decisions  which  set  the  size  of  the  computing  capacity  of  a 
system  often  are  made  early  in  the  design  process.  Once 
made,  these  sizing  determinations  are  extremely  difficult 
to  modify  due  to  space,  weight,  power,  and  cost  considera¬ 
tions. 

3.  Does  the  operating  software  for  the  prime  system  provide 
access  to  the  operating  system  or  executive  software, 
applications  programs,  and  databases,  as  appropriate  I  to 
allow  interactive  ET  software  support?  These  software 
"hooks"  are  essential  to  the  process  of  generating 
effective  ET  programs.  This  fact  is  frequently  overlooked 
in  the  decision  process  of  generating  and  refining  the 
prime  item  operating  system  software. 

As  with  other  T&E  questions,  these  three  questions  should  be 
answered  iteratively  as  the  CE  process  is  conducted.  Early  iterations 
can  disclose  the  appropriateness  of  the  analyses  designed  to  support 
answering  these  questions .  Later  T&E  iterations  will  establish  the 
actual  capability  and  effectiveness  of  the  resulting  system  in  the  form 
of  both  system  and  human  performance. 


Issue  Number  2:  T&E  for  Sustainment  Training 


ET  can  be  applied  to  all  types  of  training.  Some  very  common 
applications  of  military  training  are  initial  skills  acquisition, 
sustainment  of  skill  levels,  skill-level  progression  training,  and 
replacement  training  during  unit  rest  and  reconstitution  in  wartime. 

ET  has  potential  application  to  all  of  these  areas,  but  it  is 
especially  suited  to  sustainment  and  other  unit  training  applications 
because  ET  is  usually  located  with  the  fielded  systems,  in  units. 

If  ET  is  Co  be  a  delivered  part  of  the  prime  system,  Chen  it 
should  be  validated.  However,  there  are  major  differences  between  '^&E 
designed  for  sustainment  and  other  unit  training  versus  initial  skii'ic 
acquisitiori  training  (that  usually  cakes  place  in  Che  institution). 
Evaluation  of  acquisition  training  is  the  basis  for  most  current  T&E 
emphasis  and  is  not  dealt  with  here.  This  subsection  deals  with  how  to 
accomplish  T&E  for  sustainment  and  other  unit  training. 
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First,  tha  ET  which  is  to  be  evaluated  must  truly  be  the  ET 
intended  for  sustainment  purposes.  Once  it  has  been  decided  that  ET 
will  be  used  for  sustainment  training,  it  is  important  to  make  a 
distinction  between  the  content  of  ET  intended  for  naive  trainees 
(e.g.,  skills  acquisition  training)  and  the  content  of  ET  intended  for 
already  trained  personnel  who  are  being  sustained.  This  distinction  is 
not  always  clear  to  the  system  developers. 

Next,  there  should  be  a  plan  for  the  investigation.  A  possible 
experimental  design  for  the  evaluation  of  ET  sustainment  training 
capability  is  shown  in  Table  1. 


Table  1 


An  Experimental  Design  for  Comparing  the  Effects  of  Sustainment 
Training  vs.  No  Sustainment  Training 


Acquisition 

Training 

Test  1 

Delay 

Test  2 

Sustainment 

Training 

Test  3 

Group  1 

X 

X 

X 

X 

X 

X 

Group  2 

X 

X 

X 

X 

Delay 

X 

This  design  includes  several  features  to  evaluate  a  number* of  aspects 
of  sustainment  training,  to  give  a  complete  understanding  of  what  has 
occurred.  There  is  a  possibility  chat  not  every  part  of  this  design 
can  be  applied  in  every  test  situation,  for  reasons  of  resource 
availability.  This  design  can  and  should  be  used  as  early  as  is 
feasible  in  the  ET  development  program  as  an  EUT&E,  conducted  in-plant 
at  Che  contractor's  facilities  (in-plant  testing  is  discussed  in 
greater  detail  under  issue  number  3).  Note  chat  many  small  evaluations 
of  elements  of  training  could  be  conducted,  using  this  design,  instead 
of  a  larger  and  resource-intensive  evaluation  of  large  amounts  of 
training . 

First,  Che  design  uses  two  groups  of  trainees  for  evaluation. 

Both  groups  are  given  acquisition  training  through  whatever  means  are 
appropriate.  Acquisition  training  is  followed  by  a  performance  test 
(Test  1).  This  test  establishes  the  level  of  performance  for  each 
group  following  acquisition  training.  Next,  both  groups  undergo  a 
delay  period  during  which  they  do  not  practice  or  have  anything  to  do 
with  Che  new  equipment.  This  decay  is  important,  because  it  is  idiat 
makes  Che  sustainment  necessary.  If  the  soldiers  were  using  the  equip¬ 
ment  regularly  there  might  be  little  or  no  reason  to  need  sustainment 
training.  However,  the  usual  real-world  situation  is  Chat  soldiers  get 
little  time  actually  using  their  equipment  as  they  would  under  wartime 
conditions.  A  reasonable  period  of  delay  is  from  two  weeks  to  two 
months,  depending  on  Che  anticipated  rate  of  decay  of  skills  to  be 
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trained  or  sustained  for  a  system.  The  sustainment  evaluation  period 
begins  with  a  second  test  (Test  2)  of  both  groups,  to  establish  their 
performance  levels  just  prior  to  sustainment  training.  Group  1  then 
receives  £T  sustainment  training;  Group  2  experiences  no  training  or 
interactions  with  the  equipment.  Then,  both  groups  are  tested  a  third 
time  (Test  3). 

To  better  understand  the  reasons  behind  each  of  these  tests,  this 
paragraph  describes  the  information  provided  by  each  test.  The 
difference  in  scores  between  Test  1  and  Test  2  shows  the  decrement  in 
performance  (after  acquisition)  due  to  lack  of  practice.  The  differ¬ 
ence  in  scores  between  Test  2  and  Test  3  for  Group  2  shows  how  much 
relearning  goes  on  just  by  experiencing  the  test  but  with  no  other 
training.  The  difference  between  Test  2  and  Test  3  for  Group  1  shows 
the  improvement  due  to  both  the  test  experience  and  sustainment 
training.  The  increased  improvement  of  Group  1  over  Group  2  (from  Test 
2  to  Test  3)  is  the  increment  attributable  to  sustainment  training 
alone . 
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There  are  several  implications  of  this  approach  to  the  support 
activities  that  must  take  place: 

1.  An  effective  acquisition  training  program  must  exist  to 
deliver  initial  training. 

2.  The  sustainment  training  program  must  be  developed  enough 
to  permit  evaluation  of  its  training  ability. 

3.  Soldiers  must  be  available  at  three  separate  times,  once  to 
receive  acquisition  training,  again  and  to  receive  sustain¬ 
ment  training  (or  simply  performance  evaluation  for  the 
control  group),  and  again  to  be  retested.  These  must  be 
the  same  soldiers. 

4.  The  trainees  must  be  equivalent  to  those  idio  will  receive 
sustainment  training  under  real  conditions. 

5.  Sufficient  soldiers  must  be  available  for  each  group. 
Ideally,  the  same  number  of  soldiers  should  be  in  each 
group.  If  a  compromise  must  be  made,  reduce  the  number  in 
Group  2. 

This  discussion  focused  only  on  sustainment  training  intended  to 
maintain  the  proficiency  of  soldiers  at  or  slightly  above  that  which 
they  achieve  following  acquisition  training.  The  development  of  higher 
levels  of  ability  ("expert-level"  training)  requires  further 
evaluation.  Expert  training  evaluation  would  require  training  soldiers 
to  a  stable  level  of  good  performance,  followed  by  application  of  the 
expert  training  program  and  further  measurement. 
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Issue  Number  3:  Performance  Testing 


Performance  testing  is  defined  as  evaluation  of  the  ET  product  as 
it  is  developed  by  performing  actual  training  tests  with  an  appropriate 
trainee  sample  and  breadboard,  brassboard,  or  prototype  ET  subsystems. 
Testing  sustainment  training,  Issue  Number  2,  is  one  subset  of 
performance  testing.  In  most  cases,  the  ET  component  test  article  will 
be  in  the  form  of  preliminary  or  experimental  configurations  of 
hardware  and  software  developed  at  the  in-plant  location.  This  implies 
that  performance  testing  will  most  often  be  implemented  as  in-plant 
testing  throughout  the  evolutionary  stages  of  development,  as  a  part  of 
Technical  Test  (TT)  and  EUT&E. 


In-Plant  Testing 

In-plant  testing  of  training  capabilities,  in  the  sense  of 
formative  and  summative  evaluations  of  training  effectiveness  during 
prime  system  development,  is  not  officially  called  for  in  any  of  the 
DoD  Directives,  Standards,  Army  Regulations  or  Pamphlets.  Formative 
evaluation  involves  review  and  testing  of  individual  pieces  of  ET 
hardware  and  software  or  ET  courseware  during  the  development  phases. 
Summative  evaluation  is  done  when  all  the  pieces  can  be  combined  into 
an  operating  whole,  and  assesses  the  levels  of  skill  or  performance 
resulting  from  training.  Because  of  the  unique  nature  of  the  design  of 
ET,  testing  must  occur  in  the  early  phases  of  development  and,  there¬ 
fore,  must  be  done  "in-plant"  as  the  prime  system  and  the  ET  component 
evolve.  Such  testing  must  also  be  done  in  coordination  vith  the  devel¬ 
opment  and  testing  of  the  end  item  equipment,  as  described  for  CE,  to 
assure  that  deficiencies  can  be  remedied  while  preserving  the  operating 
objectives  of  both  the  prime  system  and  ET. 

Two  kinds  of  testing  are  required:  1)  testing  to  confirm  that  the 
ET  component  operates  as  expected  (do  the  hardware  and  software  and  the 
instructional  features  function?),  and  2)  to  confirm  that  training 
presented  via  the  ET  subsystem  (including  courseware)  teaches  appropri¬ 
ately  (does  the  ET  subsystem  meet  learning  objectives?).  Following  the 
CE  concept,  the  "does  it  function"  question  pertaining  to  ET  subsystem 
components  can  be  examined  on  an  individual  or  subset  basis  as  designs 
are  transformed  to  brassboard  or  prototype  fabrication.  To  examine 
whether  ET-based  training  teaches,  an  operating  ET  subsystem  or  operat¬ 
ing  element  of  the  complete  ET  subsystem  is  necessary,  along  with 
representative  courseware  which  would  be  presented  on  or  by  that 
element , 

Results  from  these  two  types  of  tests  should  influence  both 
continuing  ET  component  design  and,  in  all  probability,  continuing 
prime  item  system  design,  as  well.  In  addition,  results  from  the  test 
of  whether  ET  teaches  should  continue  to  structure  the  specific  nature 
and  content  of  the  developing  courseware. 
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Appropriate  ET  test  and  evaluation,  from  the  emergence  of  the 
earliest  ET  design  through  system  fielding,  requires  significant 
amounts  T&E  time  and  resources.  To  do  ET  T&E  will  require  the 
soldier  cerface,  or  functioning  portions  of  it,  plus  the  appropriate 
prime  system  software  drivers,  databases,  and  simulation  displays,  as 
well  as  ET- implementing  software  and  (if  applicable)  ET  courseware  and 
ET-specific  hardware.  In  many,  if  not  most  instances,  required  T&E 
periods  will  exceed  the  amount  of  time  that  can  be  reasonably  made 
available  on  engineering  breadboards,  brassboards,  or  prototypes.  This 
will  be  especially  true  for  the  assessment  of  courseware,  since  studies 
with  actual  training  exercises  and  multiple  students  (subjects)  will  be 
necessary.  Such  testing  will  not  only  require  access  to  an  appropriate 
facility  (a  working  ET  interface),  but  will  also  necessitate  an  ongoing 
program  of  training  material  development. 

It  is  strongly  recommended  that  the  Request  for  Proposals  (RFP) 
for  every  prime  item  system  for  which  ET  is  a  principal  consideration 
require  an  ET  testbed.  This  testbed,  or  series  of  testbeds,  will 
permit  the  continuous  formative  and  summative  evaluation  of  ET  as  the 
system  evolves,  without  interfering  with  the  ongoing  engineering 
design,  testing,  and  fabrication  of  the  prime  system.  There  may  be 
specific  instances  vAiere  ET  testing  can  be  effectively  done  using 
testbeds  designed  for  other  purposes.  As  previously  stated,  the 
results  from  such  testing,  done  in  parallel  with  engineering  develop¬ 
ment,  will  provide  direct  input  not  only  to  the  design  of  the  ET 
subsystem,  but  to  the  overall  design  of  the  prime  item  system,  and  will 
allow  insightful  and  cost  effective  tradeoffs  between  subsystem  designs 
to  be  made  and  further  evaluated. 


Testable  ET  Segments 

Successful  ET  development  depends  upon  the  following  major 
activities: 

1.  Establishing  a  timely  working  relationship  between  training 
developers  and  other  members  of  the  prime  item  system 
design  team. 

2.  Defining  the  role  of  effective  ET  in  both  the  total  train¬ 
ing  system  and  in  the  context  of  job  performance  (i.e., 
identifying  which  skills  and  knowledges  will  be  a  part  of 
ET  for  individuals,  crews,  or  netted  exercises). 

(These  first  two  activities  are  not  traditional  T&E 
functions.  However,  each  is  essential  to  effective 
integration  of  ET  with  the  prime  equipment.  These 
activities  are  discussed  more  fully  in  Appendix  B.) 

3.  Integrating  requirements  of  the  ET  component  design  with 
those  for  the  prime  item  system  (i.e.,  computer  capacity, 
soldier-Tnachine  interface,  software  "hooks,"  etc.). 
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4.  Configuring  the  ET  component  interface  so  that  interactive 
training  can  occur. 

5.  Developing  courseware  and  associated  software  to  achieve  ET 
component  goals. 

Items  3-5  should  be  identified  and  cited  as  testable  items  in  the  RFP 
and  contract.  Appropriate  planning  will  permit  the  individual  testing 
of  these  products  as  a  part  of  planned  CE  and  will  foster  ET  materials 
development  in  testable  segments.  When  specified  as  contract  deliver¬ 
ables  (perhaps  as  Data  Items  under  a  training  contract  line  item, 
supporting  specific  performance  requirements  included  in  the  system 
specification  document  or  Statement  of  Work),  plans,  progress,  and  T&E 
results  will  be  reportable  as  part  of  the  In-Process  Reviews  (IPRs). 
This  type  of  developmental  exposure  facilitates  coordination  and  the 
resolution  of  trade-off  and  design  differences. 


T&E  Courseware  Which  Supports  Operational  Testing 

Once  an  operational  interface  is  available  in  the  form  of  an  ET 
testbed,  many  of  the  tests  which  will  be  conducted  will  require  actual 
or  representative  courseware.  As  discussed,  this  courseware  will 
permit  the  assessment  of  the  functionality  of  the  ET  hardware  and 
operating  software,  and  of  the  effectiveness  of  the  intended  courseware 
configuration  (Do  trainees  learn  and  retain  the  appropriate  skills  and 
knowledges?).  Often,  implementing  these  tests  will  require  developing 
specialized  courseware  designed  specifically  to  activate  a  particular 
ET  component  or  function  in  a  selected  way.  In  ocher  instances, 
courseware  will  be  needed  to  assess  Che  viability  of  a  particular 
approach  to  training,  or  the  application  of  a  specific  ET  medium  (e.g., 
simulation  or  stimulation  involving  prime  item  SSI  displays).  For 
example,  testing  may  require  a  unique  piece  of  courseware  to  structure 
a  test  of  whether  Che  timing  of  a  specific  display  generation  fits  a 
particular  practice  scenario  or  situation.  In  both  instances,  the 
courseware  may  not  have  direct  application  Co  some  ultimate  sec  of  user 
training  requirements  (since,  for  one  thing,  these  requirements  may  not 
be  completely  known  at  Che  stage  of  development  when  such  testing  must 
be  initiated).  Often,  the  courseware  developed  to  support  such  testing 
may  not  be  useful  beyond  its  T&E  function.  However,  if  Che  courseware 
developers  stay  in  close  contact  with  Che  TIWG  and  combat  developers, 
it  will  be  possible  in  many  instances  Co  configure  test  courseware  to 
also  provide  direct  training  support  to  Initial  User  Test  and 
Evaluation  (lUT&E)  and  Follow-on  Test  and  Evaluation  (FOT&E),  even 
though  Chat  courseware  may  not  be  ^ac  is  ultimately  fielded. 
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APPENDIX  A 


AMC/TRADOC  PAMPHLET  70-2,  MATERIEL  ACQUISITION 
HANDBOOK  (1987) 

CHAPTER  13,  TEST  AND  EVALUATION 
Chapter  Guide 


This  chapter  describes  test  and  evaluation  (T&E)  procedures  in 
terms  of  the  Army  Streamlined  Acquisition  Process  (ASAP).  Information 
comparing  activities  in  the  DOD  and  ASAP  models  is  provided  in  Chapter 
1  and  Appendix  j. 


Chapter  Proponent  Offices 


AMC:  AMCQA-ST 

TRADOC :  ATCD-TP 


References 


DOD:  DODD  5000.3-M-l 

DA:  AR  15-18 

AR  70-1 
AR  70-10 
AR  71-3 
AR  71-9 
AR  702-3 
AR  702-9 
PAM  70-21 
PAM  71-3 

TRADOC:  PAM  71-13 
PAM  71-15 


Objectives 


T&E  is  conducted  to  assist  the  decisionmakers  in  reducing  and 
assessing  acquisition  risks  by: 

a.  Verifying  attainment  of  technical  performance 

specifications,  objectives,  and  supportab ility  to  include 
logistics  supportability. 
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b.  Verifying  materiel  defect. 


c.  Assessing  operational  effectiveness,  operational 
suitability,  and  readines.s. 

d.  Determining  training  requirements,  compatibility,  and 
interoperability  among  Army  systems  and  with  the  North 
Atlantic  Treaty  Organization  and  other  Services. 


Overview/ Responsibilities 

T&E  are  integral  facets  of  the  materiel  acquisition  process.  T&E 
ultimately  provides  the  data  to  answer  the  basic  concerns  of  (1)  will 
the  systems  perform  like  it's  supposed  to,  (2)  can  the  soldier  use  it, 
and  (3)  can  we  afford  it? 

Army  policy  calls  for  integrated  testing  where  feasible  and  using 
all  available  data  (e.g.,  contractor,  other  Services,  and  foreign)  for 
evaluation.  This  policy  is  aimed  directly  at  reducing  testing  by  using 
all  available  data  for  planning  and  evaluation.  The  two  basic 
categories  of  test  occurring  throughout  the  materiel  acquisition 
process  described  in  AR  70-10  are  Technical  Test  (TT)  and  User  Test 
(UT).  TT  determines  the  engineering,  safety,  and  manufacturing  aspects 
of  the  equipment.  UT  determines  the  troop  acceptability  aspects, 
opeiational  effectiveness,  and  suitability  in  the  tactical  environment, 
The  technical  tester  is  TECOM;  the  technical  independent  evaluator 
(TIE)  is  either  TECOM  or  AMSAA,  as  designated  by  HQ  AMC.  The 
operational  tester  and  evaluator  is  either  TRAJDOC  or  the  Operational 
Test  and  Evaluation  Agency  (OTEA),  designated  by  ODCSOPS.  TRADOC  also 
performs  UT  and  experimentation. 

The  acquisition  process  initial  actions  occur  when  the  user 
requirement  begins  formulation  through  the  MAA/BDP  process  at  TRADOC. 
The  Mission  Area  Analysis  Test  Advisory  Group  (MAATAG)  identifies  and 
plans  early  TRADOC  experimentation  and  test  using  standard  and 
surrogate  equipment.  These  experiments,  coupled  with  the  materiel 
developer  (MATDEV)  technology  experimentation/demonstration,  provide 
the  necessary  information  for  program  formulation  at  the  Technology 
Integration  Steering  Committee  (TISC)  I  proceedings. 

If  the  alternative  of  developing  a  weapon  system  to  satisfy  the 
user's  deficiency  is  selected,  TISC  I  triggers  a  system  development 
program  start.  At  this  point,  the  acquisition  team  representatives 
meet  and  form  the  TRADOC  chaired  Spec ial/ Joint  Working  Group  (SWG/ 

JWG) .  The  MATDEV  assumes  responsibility  for  materiel  system  in  about 
60  days  from  the  first  meeting  of  the  SWG.  TRADOC  continues  with  the 
responsibility  of  doctrine,  tactics,  organization,  training,  logistics, 
etc . 
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Critical  evaluation  issues  are  prepared  by  the  combat  developer 
(CBTDEV)  in  coordination  with  the  MATDEV,  TIE,  and  operational 
independent  evaluator  through  the  SWG/JWG.  These  issues  accompany  the 
O&O  Plan  until  it's  approved.  Critical  issues  will  be  updated  and 
approved  with  ROC  development.  The  CBTDEV  and  operational  evaluator 
supporting  issues  are  reviewed  by  the  TMEC. 

The  TIE  and  developer's  supporting  issues  are  reviewed/ approved  by 
the  AMC  Materiel  Acquisition  Review  Board  (MARB).  The  format  for 
submission  of  critical  evaluation  issues  and  criteria  is  as  shown: 

a.  ISSUE:  Questions  applicable  to  evaluating  operational 
effectiveness  and  suitability  of  a  system. 

b.  SCOPE:  Conditions  applicable  to  the  issue. 

c.  CRITERIA:  Quantitative  measures  of  the  system's 
operational  effectiveness  and  suitability  used  to  judge 
whether  the  system  satisfies  the  issue. 

d.  RATIONALE:  Justification  for  the  criteria. 

It  is  the  responsibility  of  the  acquisition  team  to  integrate/ 
combine  tests  for  the  most  efficient  and  cost-effective  test  program. 
This  is  done  through  the  Test  Integration  Work  Group  (TIWG). 

A  TIWG  is  established  by  MATDEV  based  upon  the  draft  O&O  Plan. 

The  TIWG  is  composed  of  the  MATDEV  (chair),  CBTDEV,  technical  tester, 
TIE,  operational  tester,  operational  independent  evaluator, 
logistician,  and  trainer.  Other  representatives  may  be  added,  as 
necessary . 


NOTE 

Initially  the  TIWG  membership  is  essentially  the 
same  as  the  SWG/JWG  and  are  usually  the  same  faces. 


Organizations 


w 

Reps 

Product 
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The  TIWG  develops  a  Test  and  Evaluation  Master  Plan  (TEMP)  that 
covers  all  T&E  through  the  production/ deployment  phase.  In  short,  It 
provides  a  road  map  of  T&E  through  the  acquisition  process  for  each 
system/item.  Because  of  the  update  procedures,  it  is  a  ’’living" 
document.  TEMP  procedures  and  formats  are  contained  in  AR  70-1  and  DA 
Pamphlet  70-21.  By  virtue  of  the  TEMP  format,  the  TIWG  is  forced  to 
use  a  systematic  approach  that  identifies  what  data  is  needed  (Master 
Evaluation  Plan  (MEP))  to  satisfy  the  issues  before  planning  tests  to 
produce  data.  Information  contained  in  the  TEMP  will  be  used  as 
structuring  contractual  documentation.  The  TEMP  is  submitted  as  a 
package  with  the  AS  for  approval  by  the  appropriate  level  MADP  before 
entering  Proof  of  Principle. 

For  the  responsible  MATDEV,  the  TEMP  provides  a  documented  log  of 
plans,  coordinations,  and  results.  For  all  others,  it  provides  a 
documented  agreement  of  what  is  planned  and  has  been  accomplished. 

Since  the  TEMP  is  a  living  document,  an  ■'ngredient  for  feedback  is 
necessary.  This  ingredient  is  the  common  T&E  database.  This  mechanism 
will  permit  the  acquisition  team  to  "continuously  evaluate/analyze 
functionally"  system  progress.  Systematic  review  of  the  information  in 
the  common  T&E  database  by  the  acquisition  team  exposes  problems/issues 
for  early  solutions  at  the  working  level,  and  encourages  efficient 
system  development.  Other  considerations  during  test  planning  are: 

T&E  Managers,  Test  Technology/Faci llties ,  Test  Funding,  and  Continuous 
Evaluation  (CE). 


Test  and  Evaluation  Managers 


Each  AMC  MSC  has  designated  a  test  manager  to  manage  the  T&E 
functional  area  within  the  command.  The  test  managers  coordinate 
directly  with  HQ  AMC,  TECOM,  and  other  test  managers  and  assist  test 
planners  as  a  matrix  function. 


Test  Technology/Facilities 


With  the  advanced  technology  that  is  being  applied  to  new  weapon 
systems,  the  need  for  appropriate  test  technology  to  support  testing  of 
these  systems  is  recognized.  TECOM  has  established  a  central 
coordination  office  to  ensure  that  all  AMC/Army  testers  are  aware  of 
new  test  technology  being  developed/built.  The  central  coordinator 
assists  in  eliminating  duplicate  developmant  efforts  and  Ciaxlmlzlng  the 
use  of  existing  test  facilities  by  sharing  information  on  test 
facilities.  TECOM  Technology  Office  -  AMSTE-TC,  APG,  MD. 
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Test  Funding 


MATDEVs  are  responsible  for  programing  funds  for  all  system  test 
costs  lAW  AR  37--100,  AR  70-10,  AR  71-3,  and  AMC  Supplement  to  AR 
37-100.  Users  are  responsible  for  programing  funds  for  all  innovative, 
CEP,  FDTE,  and  OTs  using  a  system  lAW  AR  71-3.  NOTE:  MATDEV  funds 
materiel  portion  of  OT. 


Continuous  Evaluation  (CE) 


CE  represents  a  thrust  to  a.ssure  the  continuous  flow  of  updated 
information  regarding  system  status  including  planning,  testing,  data 
compilation,  analysis,  evaluation,  and  conclusions,  and  is  available  to 
all  members  of  the  acquisition  team  plus  decisionmakers  from  the 
initial  O&O  Plan  through  deployment  and  assessment  of  field 
performance.  CE  is  performed  by  each  member  of  the  acquisition  team. 

A  major  objective  is  tor  the  members  tc  be  active  in  surfacing  critical 
problems  at  the  earliest  opportunity  so  that  they  may  be  addressed  and 
resolved  before  they  impact  important  decisions.  This  helps  eliminate 
last  minute  surprises. 

CE  essentially  ensures  that: 

a.  Testers  and  independent  evaluators  are  exposed  to  the 
system  early  so  that  realistic  T&E  requirements  may  be 
planned  and  developed. 

b.  Requirements  and  specifications  that  drive  T&E 
considerations  are  made  available  to  the  testers  and 
evaluators  to  stabilize  the  test  program. 

c.  Contractor  and  Government  test  data  as  well  as  data  from 
other  sources  are  made  available  to  be  used  in  the 
continuous  evaluation  process. 

d.  Optimum  testing  is  scheduled  to  preclude  duplication  in 
the  interests  of  reducing  testing  for  the  overall  test 
program. 

The  evaluators  assess  and  evaluate  the  technical  performance, 
operational  effectiveness,  and  operational  suitability  of  a  system 
throughout  the  entire  materiel  life  cycle.  The  results  of  each  of 
these  evaluations  are  provided  by  each  evaluator  to  the  Acquisition 
Executives.  Within  AMC,  TECOM  and  AMSAA  provide  TIEs;  the  MATDEV  (PH, 
Team  Manager,  et  al.)  provides  a  developer's  evaluation.  Hie 
operational  independent  evaluators  are  either  Ol'EA  or  TRADOC.  TECOM  is 
the  repository  for  all  I&E  data  and  the  database  is  located  in  APG  MD 
The  test  community  feeds  the  database  with  test  data  per  the  19Bb  MOU 
between  TRADOC,  OTEA,  and  AMC. 
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Time  Constraints 


Once  a  system  has  been  identified  to  be  developed,  the  acquisition 
team  representatives  are  designated.  The  O&O  Plan,  AS,  and  T^P  are 
developed.  Note  the  reliability  plan  must  be  developed  before  the  TEMP 
can  be  finalized  as  a  coordinated  draft.  Also,  the  T^P  must  be 
approved  by  the  appropriate  level  MADP  before  any  testing  occurs  In  the 
Proof  of  Principle  Phase.  The  goal  Is  to  provide  test  results  and 
evaluation  analyses  on  production  items  for  Milestone  III.  Approved 
test  waivers  are  covered  in  AR  70-10. 


Process  Outline 


Detailed  procedures  for  T&E  are  described  on  the  following  pages 
in  the  form  of  descriptive  paragraphs  keyed  to  corresponding  actions  on 
the  charts.  The  format  displays  specific  activities  in  each  phase  of 
■he  materiel  life  cycle  conducted  by  the  CBTDEV,  MATDEV,  evaluators, 
testers,  and  industry.  It  should  be  recognized  that  the  charts  depict 
the  product  (document,  activity,  etc.)  under  responsible  activity. 
Activities  are  plctorally  ordered  to  provide  a  general  time  sequence. 

1.  MAA/BDP  are  the  Initial  materiel  acquisition  process 
actions . 

2.  Technology  experimentation/ demonstration  occurs  during 
early  technology  feasibility  research. 

3.  User  Experimentation.  .  TRADOC  user  testers  perform  user 
experimentation  (CEP,  FDTE,  or  innovative  testing)  as 
outlined  by  the  MAATAG,  CBTDEV,  and  training  developer. 
These  early  and  continual  user  test  and  experimentation 
efforts  serve  to — solidify  the  materiel  need  and  support 
O&O  Plan  preparation,  support  development  of  critical 
evaluation  Issues  and  criteria,  and  foster  initiation  of 
doctrine,  tactics,  organization,  and  training  packages. 
Experimentation  planning  and  reporting  guidance  is 
contained  in  AR  71-9. 

A.  MAATAG.  The  MAATAG  functions  as  a  subcommittee  of  the 
MAA.  It  identifies  user  experimentation  requirements  to 
support  the  MAA  and  operational  requirements  definition 
process.  Also,  the  MAATAG  defines  the  experiment  Issues 
and  provides  initial  planning  guidance. 

5.  MAMP.  See  chapter  2. 

6.  Safety  Release.  A  safety  release,  if  necessary,  will  be 
provided  by  TECOM  as  a  customer  service  lAW  AR  350-16. 


7.  The  Technology  Integration  Steering  Commit  ee  (TISC)  pairs 
technological  opportunities  with  emerging  ii:4airement.s. 

It  includes  joint  (MATDEV/CBTDEV)  representation  and  meets 
semi-annually.  TISC-I  matches  technological  opportunities 
with  Army  thrusts  and  emerging  mission  needs.  It  triggers 
the  preparation  of  an  O&O  Plan  or  JMSNS.  It  also  directs 
technology  maturation  actions  needed  for  subsequent  steps. 
TISC  findings  support  MAMP  and  LRRDAP  prioritization  of 
resources,  as  well  as  MATDEV/CBTDEV  MARB  deliberat ions/In- 
Process  Reviews  (IPRs). 

8.  The  O&O  Plan  (or  JMSNS,  if  required)  is  the  program 
(system)  initiation  document  and  is  approved  before  TISC 
II  review. 

9.  Critical  evaluation  issues  and  criteria  will  be  approved 
by  the  Materiel  Acquisition  Decision  Process  (MADP)  Review 
and  included  in  the  TEMP.  These  issues  lead  to  the  test 
issues  through  the  Independent  Evaluation  Plan  (lEP) 
process . 

10.  The  TIWG  formulates  the  first  broad  scope  TEMP  which 
includes  a  reliability  plan  (AR  702-3).  This  group 
ensures  the  testing  program  outlined  by  the  TEMP  is 
coordinated  in  support  of  the  AS.  All  testing  will  be 
identified  in  the  TEMP,  to  include  testing  of  product 
improvements.  Ihe  TEMP  is  a  "living"  document.  Between 
milestones,  this  coordinated  TEMP  is  considered  approved 
by  TIWG  consensus  unless  the  MADP  decision  authority  (vho 
is  notified  in  writing  of  changes)  disapproves.  Each  TIWG 
representative  is  responsible  for  his  command's 
concurrence.  The  overall  TEMP  requires  concurrence  prior 
to  milestones  by  all  TIWG  membership.  It  is  forwarded  to 
the  MADP  review  body  for  approval  as  part  of  the 
supporting  documentation.  Instructions  for  the  TEMP  and 
its  preparation  are  in  AR  70-10  and  DA  Pam  70-21. 

11.  CE  starts  after  TISC  I  and  continues  throughout  the 
process.  The  T&E  database  used  for  CE  will  be  fed  by  the 
TIWG  membership.  The  gradient  on  the  chart  indicates  the 
initial  data  input  to  the  database. 

12.  Experimentation  Reports.  These  are  the  reports  for  all 
previous  user  and  technical  experimentation  that  will  be 
encompassed  in  the  formulation  of  the  system.  This  data 
will  be  part  of  the  initial  TEMP  data  reviewed  by  the 
TISC. 


13.  Outline  Test  Plans  (OTP}/Resuine  Sheets  are  prepared  lAW 
TRADOC  Pam  71-15. 


NOTE 

The  OTP  is  updated  every  6  months  for  Test  Schedule 
and  Review  Committee  (TSARC)  for  FDTE  resume  sheets 
are  for  CEP  and  are  approved  by  TRAOOC  with  changes 
worked  out  between  proponent  and  tester.  The  OTP  is 
a  resource  document  which  usually  is  prepared  for 
the  TSARC  (AR  15-38)  by  the  operational  tester. 

OTPs  will  be  prepared  by  the  MATDEV  when  additional 
testing  resources  (normally  FORSCOM  support)  are 
required.  It  contains  a  listing  of  the  necessary 
resources  and  administrative  information  required 
for  support  of  a  test.  The  OTP  also  contains  the 
critical  test  issuei  test  conditions,  and  scope. 
Additionally,  the  OT  OTP  will  address  tactical 
context . 

14.  The  TIE  and  operational  independent  evaluator  each  prepare 
an  Independent  Evaluation  Report  (lER)  for  all  aspects  of 
evaluation  responsibilities  relative  to  the  system  includ¬ 
ing  market  investigation.  The  lEP  details  the  independent 
evaluator's  actions  for  the  evaluation  of  the  system.  It 
is  periodically  updated,  at  least  annually,  reflecting 
materiel  and  program  changes.  The  operational  independent 
evaluator  has  the  responsibility  for  preparation  of  the 
Master  Evaluation  Plan  (MEP).  The  NEP  consolidates  the 
technical  and  operational  independent  evaluation  plans 
with  the  materiel  developer's  (MSC/PM)  detailed  plans  for 
evaluation  of  the  system.  The  evaluation  plans  take  into 
account  all  available  data  to  preclude  unnecessary  testing 
while  assuring  that  evaluation  objectives  are  achieved. 

The  MEP  will  identify  each  issue  for  evaluation  and  the 
methodology  to  be  used.  The  MEP  will  specify  the 
procedures  for  exchange  of  evaluation  information,  if 
necessary.  The  MEP  requires  concurrence  by  member 
agencies  of  the  TIWG.  The  MEP  will  be  an  annex  to  the 
TEMP.  Instructions  for  its  preparation  are  in  DA  Pam 
70-21.  The  Test  Design  Plan  (TDP)  is  a  formal  document 
which  supports  the  TEMP  and  may  be  provided  as  an  annex  to 
the  TEMP.  TDPs  are  derived  from  the  lEP  and  are  prepared 
by  the  TIE  and  the  operational  tester.  The  TDP  is 
responsive  to  the  technical  and  operational  issues 
developed  by  each  evaluator.  It  includes  a  complete  test 
design,  description  of  required  tests,  the  conditions 
under  which  the  system  is  to  be  tested,  a  statement  of 
test  criteria,  and  measures  and  plans  for  data  collection 
for  the  TDP. 
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15.  System  Support  Package  (SSP).  The  list  of  Items  In  the 
SSP  (plus  shortages)  to  test  is  aeveloped  by  the  MATDEV. 
SSP  for  each  test  Is  developed  lAW  DA  Pam  71-3  and  refined 
before  each  readiness  review  and  test. 

16.  Test  Support  Package  (TSP).  TRADOC  combat  developers  and 
training  developers  begin  preparation  of  the  following  TSP 
in  preparation  for  experimentation  during  the  Proof  of 
Principle  Phase  and  OT  during  the  Development  Proveout 
Phase: 

a.  Doctrinal  and  Organizational  Test  Support  Package. 

b.  Threat  Support  Package. 

c.  Training  Test  Support  Package. 

These  packages  are  refined  as  the  program  progresses  lAW 
DA  Pam  71-3. 

17.  TISC  II  reviews  the  "match”  and  maturity  of  TISC  I 
solutions  for  suitability  to  advance  to  Proof  of 
Principle.  The  MARB  follows. 

18.  MARB.  See  MADP  chapter  15. 

19.  A  "Star  Review"  ("16  Star  Review"  for  major  programs)  is 
held  to  establish  top-down  consensus  on  basic  program 
direction  at  onset  of  the  Proof  of  Principle  Phase.  The 
principles  are  determined  by  the  level  of  the  MADP 
decision  authority  (e.g.,  major  programs  -  VCSA,  USofA, 
AMCCG,  and  TRADOC  CG).  The  review  verifies  and  lends 
senior  leadership  impetus  to  the  basic  program  parameters 
and  course  of  action  outlined  for  this  acquisition  phase. 
It  reviews  the  TEMP,  AS,  and  O&O  Plan  as  a  minimum. 

20.  Update  TEMP. 

21.  Update  lEP/TDP/MEP. 

22.  ROC  is  initiated  lAW  AR  71-9.  Issues  and  criteria  are 
updated,  approved,  and  TEMP  so  updated.  ROC  is  approved 
approximately  1  year  before  Milestone  I/II. 

23.  Technical  Test  Readiness  Review  (TTRR)  is  chaired  by  the 
MATDEV  with  the  principle  TIWG  members  in  attendance  lAW 
DA  Pam  70-21  (similar  to  Operational  Test  Readiness  Review 
(OTRR)). 

24.  User  Test  Readiness  Review  (UTRR)  conducted  by  the  user 
tester  for  user  experiments  at  system  level.  The  UTRR  is 
conducted  in  the  same  manner  as  the  OTRR  described  in  the 
next  phase  (//42). 


A-9 


A-10 


25.  Contractor  begins  fabrication  of  prototype/surrogate/ 
brassboard  hardware  and  conducts  contract  required  tests. 

26.  T&E  is  conducted  in  this  phase  as  integrated  by  the  TIWG 
process  in  the  TEMP  document.  It  includes  separate, 
combined,  and  concurrent  user  experimentation  and 
technical  testing.  T&E  conducted  during  this  phase 
supports  the  hardware/software  design  through  an  approach 
tdiich  will  be  performed  at  the  component,  subsysttnn,  and 
system  level;  identifies  the  preferred  technical  approach, 
including  the  identification  of  technical  risks  and 
feasible  solutions;  examines  the  operational  aspects  of 
the  selected  alternative  technical  approaches,  estimates 
the  potential  operational  effectiveness  and  suitability  of 
candidate  systems;  supports  the  product  improvement 
proposal  (PIP)  process;  reduces  design  risks;  establishes 
contractual  compliance  including  component  qualification; 
provides  data  for  required  readiness  for  test  reviews;  and 
evaluates  technical  and  operational  issues. 

27.  A  Safety  Release  is  processed  lAW  AR  385-16  in  preparation 
for  UT. 

28.  Test  Incident  Report  (TIR)  Summary.  The  TIR  Summary  is 
prepared  by  the  MATDEV  for  the  correct  action  review.  All 
user  and  technical  test  TIRs  arc  included. 

29.  The  Test  Report  (TR)  is  a  formal  document  of  record  vhich 
reports  the  data  and  information  obtained  from  the  conduct 
of  the  test  and  describes  the  conditions  which  actually 
prevailed  during  test  execution  and  data  collection. 
Included  in  the  TR  is  an  audit  trail  of  deviations  from 
the  TDP.  TRs  are  prepared,  approved,  and  published  by  the 
technical  and  operational  testers. 

30.  A  corrective  action  review  process  conducted  by  the  MATDEV 
is  in  the  formal  process  to  assess  and  improve 
reliability,  availability,  and  maintainability  (RAM) 
performance  (See  AR  70-10).  Each  activity,  which  has  been 
assigned  responsibility  for  corrective  action  lAW  AR  702-3 
will  report  on  the  actions  that  have  been  taken  to  correct 
each  failure  mode  identified  during  tests.  Adequacy  of 
failure  analysis,  appropriateness  of  corrective  action, 
demonstration  of  corrective  action  by  tests,  verification 
of  future  implementation,  and  evaluation  of  effectiveness 
of  the  corrective  action  will  be  assessed  by  the  CBTDEV, 
MATDEV,  the  technical  and  operational  testers,  and 
evaluators.  An  audit  trail  of  changes  will  be  established 
and  reported  to  the  design  authority.  In  addition,  the 
MATDEV  (MSCs/PMs)  will  use  the  RAM-scored  data  iti  the 
development  of  reliability  and  maintainability  growth 
models  and  assess  the  impact  of  meeting  the  technical  and 
operational  thresholds. 
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31.  The  lER  is  an  independent  evaluation  of  the  system  based 
on  test  data,  test  reports,  studies,  and  other  appropriate 
sources.  lERs  are  prepared,  approved,  and  published  by 
the  technical  and  operational  independent  evaluators  at 
key  milestone  events.  Under  the  CE  concept,  the 
independent  evaluators  will  periodically  update  their 
evaluation  of  the  system.  The  lER,  a  formal  document  of 
record,  contains  an  assessment  of  the  issues  contained  in 
the  lEP  and  other  issues  as  appropriate;  the  independent 
evaluator's  conclusions;  evaluation  of  test  issues;  the 
evaluator's  position  on  the  future  capability  of  the 
system  to  fulfill  the  approved  requirements;  and  identi¬ 
fies  program  constraints  and  their  impact  on  the  evalua¬ 
tion.  The  lERs  are  provided  to  the  MAUP  review,  as 
appropriate;  the  independent  evaluator's  conclusions; 
evaluation  of  test  issues;  the  evaluator's  position  on  the 
future  capability  of  the  system  to  fulfill  the  approved 
requirements;  and  identifies  program  constraints  and  their 
impact  on  the  evaluation.  The  lERs  are  provided  to  the 
MADP  review  body.  lERs  are  normally  briefed  by  the  TIE 
directly  to  the  pre-ASARC  or  IPR;  and  by  the  operational 
independent  evaluator  directly  to  the  ASARC,  or  IPR. 

32.  Update  TEMP  for  next  phase. 

33.  Update  lEP/TDP/MEP  for  next  phase. 

34.  Update  OTP  for  FDTE  and  lOT&E  for  next  phase. 

35.  Milestone  I/II  Review. 

36.  Update  TEMP. 

37.  Update  lEP/TDP/MEP. 

38.  Contractor  builds  hardware/software  system,  conducts 
contractually  required  tests. 

39.  Instructor  and  Key  Personnel  (I&KP)  is  provided  by  the 
HATDEV  based  on  the  training  requirements  developed  by 
TRADOC  for  the  testers. 

40.  Operational  Test  Readiness  Statements  (OTRS)  are  provided 
by  the  trainer,  CBTDEV,  and  MATDEV. 

41.  Same  as  item  23. 

42.  OTRR  is  a  review  to  identify  problems  which  may  impact  the 
conduct  of  an  OT&E.  An  OTRR  is  conducted  to  determine 
changes  required  in  planning,  resources,  or  testing 
necessary  to  proceed  with  a  specific  OT.  OTRR 
participants  include  the  operational  tester  (chair), 
independent  evaluator,  MATDEV,  user  representative, 
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logistician,  HQDA  staff  elements,  and  others  as  necessary. 
The  OTRR  examines  the  OTRS  and  the  safety  release  prior  to 
the  start  of  the  test  and  may  include  review  of  initial  or 
pilot  tests.  The  OTRR  is  mandatory  for  major  and  DAP 
systems  with  the  results  reported  to  ASARC  principals.  An 
OTRR  is  conducted  for  other  systems,  as  determined 
necessary  by  the  operational  tester. 

43.  The  integrated  testing  in  this  phase  includes  FDTE,  lO&E, 
DT&E,  and  qualification  testing.  T&E  conducted  during 
this  phase  matures  development  prototype  hardware/sof t- 
ware;  provides  a  valid  estimate  of  the  system's 
operational  effectiveness  and  suitability  (including 
performance,  survivability,  reliability,  availability  £uid 
maintainability,  safety,  MANPRINT,  and  logistic 
supportability) ;  ascertains  whether  engineering  is 
complete;  identifies  design  problems  and  ascertains  that 
solutions  to  these  problems  are  in  hand;  supports  the  PIP 
process;  reduces  design  risks;  establishes  contractual 
compliance;  validates  general  and  detailed  specifications, 
standards,  and  drawings  for  use  to  procure  units  of 
products;  provides  data  for  required  readiness  for  test 
reviews;  and  evaluates  technical  and  operational  issues. 
Testing  includes  hard-tool  prototypes  to  resolve  OT 
issues.  FDTE  conducted  in  this  phase  refines  and 
validates  tactics,  organization,  and  training  before 
lOT&E. 

44.  Contractor  provides  hard-tool  prototypes  from  production 
line. 

45.  Update  TEMP. 

46.  Update  lEP/TDP/MEP. 

47.  Update  OTP. 

48.  Same  as  29. 

49.  Same  as  31. 

50.  Update  issues. 

51.  Milestone  III  review. 

52.  Update  TEMP. 

53.  Update  lEP/TDP/MEP. 

54.  Same  as  42. 

55.  Same  as  40. 
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DEVELOPMENT  PROVEOUT 


A-15 


56.  If  qualification  testing  was  not  completed  in  the 
Developtnent  Proveout  Phase,  it  will  be  completed  in  this 
phase.  Sufficient  testing  should  have  been  completed  in 
the  last  phase  to  permit  the  DOTE  to  be  reported  to 
Congress . 

57.  Same  as  31. 

58.  Same  as  29. 

59.  Preplanned  Product  Improvement  (P3I)  testing.  Testing  of 
the  first  block  improvements  to  the  system,  based  on  Proof 
of  Principle  and  Development  Proveout  Phase  testing, 
occurs  on  production  items. 

This  testing  is  integrated  by  TIWG  and  lAW  AR  70-21. 
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APPENDIX  B 


NON-TRADITIONAL  ASSESSMENT  OF  THE  ET  DESIGN  PROCESS 


Early  in  a  system  development  cycle,  testing  the  process  of  Uesign 
is  far  more  critical  then  testing  Che  products  of  design.  Prime  system 
testing  has  traditionally  examined  the  functionality  of  hardware  and 
software  against  system  specifications,  with  the  overall  concern  of 
effectively  meeting  mission  requirements.  Likewise,  training  system 
and  training  device  testing  has  also  focused  on  the  operation  of 
hardware  and  software,  and  on  the  effectiveness  of  learning  stimulated 
by  the  system  or  device.  However,  constraints  on  the  design  process 
for  ET  differ  in  a  number  of  respects  from  those  structuring  typical 
system  design  efforts.  The  most  important  of  these  constraints  is  that 
ET  and  prime  system  operational  characteristics  must  be  mutually 
supportive.  With  ET,  whether  built-in  or  hung-on,  the  design  window  is 
limited  in  time  and  rigorous  in  content.  There  is  little  opportunity 
to  recover  from  having  designed  late  or  poorly. 

The  purpose  of  this  Appendix  Is  to  call  to  the  attention  of 
contractors  and  proposal  evaluators  issues  which  appear  to  be  critical 
to  a  successful  ET  design  program.  Limited  experience  to  date  with 
development  of  ET  programs  strongly  suggests  that,  while  they  are  not 
traditional  T4E  areas,  these  issues  form  a  reliable  litmus  test  for 
probable  success. 

The  design  of  successfully  integrated  ET  is  totally  dependent  upon 
having  instructional  systems  designers  as  a  part  of  the  overall  end 
item  equipment  design  team,  and  upon  early  and  iteratively  updated 
embedded  training  requirements  analyses  which  influence  design.  With¬ 
out  a  mult idisc iplined  design  team,  and  ET-relevant  analyses,  only  luck 
can  produce  effective  ET.  With  this  level  of  criticality,  it  follows 
that  prudent  program  management  should  initiate  and  vigorously  pursue 
an  assessment  of  the  organization  and  process  for  designing;  i.e., 
"testing"  of  the  design  mechanism.  Early  on  then,  usually  well  before 
Milestone  1,  the  nature  of  the  "tests"  should  be  focused  on  assuring 
that  training  system  design,  and  the  development  of  ET  in  particular, 
will  be  appropriately  accommodated  by  developmental  planning,  organiz¬ 
ing,  and  manni.ng .  Without  the  intention  and  commitment  to  provide  the 
personnel  and  funding  resources  needed  to  attain  effective  ET,  inte¬ 
grated  ET  design  is  neither  feasible  nor  practical.  As  development 
progresses,  the  nature  of  appropriate  testing  will  shift  toward  the 
traditional  focus  of  T&E  procedures:  does  it  work,  and  does  it  meet 
stated  requirements?  Two  major  issues  are  germane  to  this  non- 
traditional  assessment  situation: 
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Issue  Number  1:  Integration  of  Embedded  Training 
With  Prime  System  Design 


Performance  testing  of  the  ET  component  generally  occurs  in 
ccnjunction  with  the  operation  of  the  prime  systen,  since  the  soldier's 
operational  interface  is  utilized  as  the  training  interface.  However, 
at  the  point  in  development  where  there  is  an  operational  soldier 
interface,  a  great  deal  of  time  and  system  design  effort  will  have  been 
expended,  and  many  design  decisions  will  have  been  firmed  in  terms  of 
configuration,  hardware  and  software  requirements,  etc.  This  may  be 
too  late  in  the  development  process  to  be  able  to  influence  system 
characteristics  to  enable  ET  functions  to  be  included  in  the  system. 

Under  the  CE  concept,  and  within  the  MANPRINT  portion  of  this  type 
of  evaluation,  the  purpose  of  testing  is  to  account  for  the  effects  of 
expected  soldier  performance,  system  ef f.’ict  iveness,  and  operational 
availability.  System  effectiveness,  in  the  case  of  ET,  refers  to  the 
effectiveness  of  the  ET  components.  Experience  teaches  that  this 
component  will  not  be  effective  unless  ET  subsystem  designers  play  a 
significant  and  active  role  along  with  the  prime  system  designers  in 
establishing  operational  parameters  and  realistic  performance  trade¬ 
offs.  Therefore,  the  ET  subsystem  must  be  planned  at  the  same  time  and 
in  concert  with  the  planning  for  the  operational  system.  This  will 
require  that  the  ET  subsystem  designers  be  an  integral  part  of  the 
design  team  from  the  very  beginning. 

At  the  time  of  preparation  of  initial  procurement  documentation, 
critical  questions  which  provide  the  criteria  for  evaluation  of 
proposals  and  subsequent  contractor  performance  include: 

1.  Is  embedded  training  being  considered  for  principal  roles 
in  the  proposed  total  training  sy.stem? 

2.  Are  instructional  technologists  (instructional  systems 
designers^  a  required  and  assigned  pr  t  of  the  design 
team? 

3.  Will  the  prime  contractor  be  selected  on  the  basis  of  his 
proposed  approach  to  training  design  (including  ET),  as 
well  as  on  his  proposed  approach  to  the  operational  system 
development? 

4.  Ic  there  a  design  philosophy  v*iich  will  consider  tradeoffs 
between  operational,  and  ET  features,  to  maximize  the 
effectiveness  of  both? 

5.  Does  the  planning  for  continuous  evaluation  include  a 
specification  which  assures  that  ET  T&E  will  be  rigorously 
pursued  in  parallel  with  the  functional  and  component 
continuous  evaluation  activities  for  the  operational 
system? 
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•>  Number  2:  Early  Identification 
Embedded  Training  Requirements 


Embedded  training  can  take  many  forms,  depending  upon  the 
characteristics  of  the  system,  the  configuration  of  the  soldier 
interface,  the  specific  skills  and  knowledges  to  be  trained,  and  the 
"stage"  of  learning  for  which  the  ET  subsystem  is  intended  (e.g., 
sustainment  as  contrasted  to  initial  acquisition).  The  process  of 
establishing  the  specific  performance  requirements  which  the  ET 
component  will  be  designed  to  support  is  the  most  critical  activity 
during  the  design  efforts  which  occur  prior  to  Milestone  I/II.  This 
criticality  is  not  different  for  ET  than  for  other  types  of  training 
system  designs;  in  all  cases  accuracy  and  completeness  are  crucial  to 
the  design  effort.  However,  in  traditional  training  system  develop¬ 
ment,  the  training  materials  which  are  developed  to  meet  operational 
needs  can  be  modified  and  augmented  at  a  later  date  in  the  development 
cycle  without  jeopardizing  the  effectiveness  of  the  system  itself,  or 
significantly  lengthening  the  duration  of  training  development.  With 
an  ET  component,  once  appropriate  tradeoffs  and  determinations  have 
been  established  to  permit  the  design  of  an  ET  interface  and  delivery 
system,  the  degrees  of  freedom  to  modify  that  design  become  severely 
restricted  as  the  operat ional  system  design  is  solidified. 

Because  it  is  essential,  to  "do  it  right  and  do  it  early,"  it  is 
important  to  assure  that  the  analyses  needed  to  establish  appropriate 
job  relevance  are  both  timely  ar.i  comprehens ’ ve .  Guideline  Volume  4 
entitled,  "Identifying  ET  Req-'it ements ,"  establishes  the  specific 
procedures  appropriate  for  generating  and  compiling  ET  requirements. 
Some  oversight  process  must  confirm  that  the  required  analyses  are 
indeed  being  accomplished,  and  that  the  data  are  sufficiently  detailed 
and  accurate  so  that  firm  design  decisions  can  be  made.  This  process 
is  probably  not  within  the  traditional  scope  of  T&E.  Nevertheless, 
training  development  members  of  or  advisors  co  the  TIWG  must  be  aware 
of  the  importance  of  this  issue  and  its  component  sub-issues. 
Overlooking  any  of  these  issues  could  prove  disastrous  from  the  point 
of  view  of  developing  well-integrated  and  effective  ET.  If  for  no 
other  reason,  training-oriented  testers  and  TIWG  members  should  be 
aware  of  these  oversight  needs,  and  attempt  to  coordinate  the  oversight 
process  with  MANPRINT  surveillance  of  the  developing  system.  Indeed, 
it  may  be  appropriate  to  include  these  issues  and  specific  training 
issues  in  the  process  of  developing  the  System  MANPRINT  Management  Plan 
(SMMP)  for  systems  anticipated  to  include  ET. 

As  with  Issue  Number  1  (see  Page  21),  the  substance  of  this  issue 
has  to  do  with  the  existence  and  the  effectiveness  of  an  analytic 
process  idiich  is  essential  to  produce  the  data  needed  for  ET  design 
'^ori'sions.  Oversight  needs  for  this  issue  deal  with  the  completeness 
anc  accuracy  of  Che  design  process ,  since  it  will  be  too  early  to 
evaluate  the  ET  design  itself.  Specific  considerations  in  the  over¬ 
sight  process  include  gathering  data  to  answer  the  following 
questions : 
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1.  Are  there  iterative  applications  of  the  guideline  proce¬ 
dures  specified  in  Volume  4  "Identifying  ET  Requirements" 
beginning  in  the  Requirements  and  Technological  Base  Phase 
in  the  Army  Streamlined  Acquisition  Process  (ASAP)  or  in 
the  Concept  Exploration  Phase  of  the  LCSMM?  Application  of 
these  procedures  should  result  in  the  identification  of 
operational  system  missions,  phases,  jobs  and  functions,  or 
tasks.  In  addition,  this  process  should  identify  signifi¬ 
cant  conditions  of  performance  lAiich  will  influence  opera¬ 
tions  and  maintenance  training  requirements. 

2.  Is  there  integration  of  the  preliminary  ET  requirements 
into  the  design  of  a  total  training  system  concept  for  the 
operational  system?  As  has  been  emphasized  throughout 
these  guidelines,  ET  can  be  but  one  component  of  the  total 
training  system  for  any  prime  operational  system.  In  order 
that  ET  maximally  support  the  total  set  of  system  training 
requirements,  it  can  only  be  designed  in  concert  with  the 
total  training  system.  At  this  point  in  the  developmental 
process,  the  existence  of  an  overall  design  effort  for  the 
complete  training  system  must  be  assessed  in  order  to 
provide  many  of  the  criteria  for  judging  the  configuration 
and  instructional  content  of  the  ET  component. 

3.  Are  there  iterative  applications  of  the  ET  requirements 
(ETR)  identification  procedures  during  the  Proof  of 
Principle  Phase  of  ASAP  (or  late  in  the  Concept  Exploration 
phase  or  early  in  the  Demonstration  and  Validation  Phase  of 
the  LCSMM)?  These  analyses  must  focus  on  the  identifica¬ 
tion  of  specific  behavioral  performance  objectives  and  on 
updating  and  expanding  the  task  identification  accomplished 
earlier.  Assessment  of  this  application  will  essentially 
consist  of  confirming  that  the  systems  approach  to  training 
(SAT)  task  analysis  procedures  are  indeed  being  accom¬ 
plished  and  are  resulting  in  a  total  task  analytic 
database. 

4.  Are  task  analytic  data  derived  from  the  ETR  analyses  being 
integrated  with  the  design  of  the  total  training  system, 
including  stand-alone  training  devices  and  schoolhouse 
training,  as  well  as  ET?  For  any  particular  prime  system 
design  effort,  it  may  be  that  various  components  of  the 
total  training  subsystem  design  are  being  accomplished  by 
different  organizations.  The  oversight  process  for  ET 
needs  to  confirm  that  the  design  of  the  ET  component  is  not 
occurring  in  isolation  from  the  design  of  the  total 
training  system  and  the  prime  itan  system. 

5.  Are  ET  requirements  being  identified  through  the  appli¬ 
cation  of  ETR  analysis  procedures  to  the  task  analysis 
data?  T.L  is  critical  at  this  phase  of  the  analysis  that 
appropriate  criteria  are  collected  to  differentiate  those 


training  requirements  tA\ich  will  be  satisfied  by  ET  from 
training  requirements  to  be  achieved  through  other  compo¬ 
nents  of  the  total  training  system.  The  oversight  process 
must  carefully  review  the  rationale  and  its  application  to 
the  task  analytic  data  for  this  differentiation.  Task 
nomination  should  be  based  on  a  careful  assessment  of 
perishability  and  criticality  criteria,  in  addition  to 
constraints  imposed  by  overall  training  system  decisions. 

6.  Is  the  decision  making  process  tdiich  reviews  the  overall 
feasibility  of  implementing  the  identified  ET  requirements 
occurring?  This  process  must  not  be  unrealistically 
constrained  by  prime  item  system  parameters  and  design 
considerations.  In  other  words,  oversight  is  appropriate 
to  assure  that  ET  emphases  have  an  appropriate  weight  in 
the  overall  prime  item  system  design  process. 
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APPENDIX  C 
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LIST  OF  ABBREVIATIONS  AND  ACRONYMS 


AMC 

AMSAA 

AP.TEP 

AS 

ASAP 

ASARC 

BDP 

CBTDEV 

CE 

CEP 

CTEA 

DCP 

DOT&E 

DSARC 

DTEA 

DTP 

ET 

EUT&E 

FDTE 

FOT&E 

I&KP 

lAW 

lEP 

lER,  lERs 

IOC 

lOE 

IPR 

lUT&E 

JDT 

JMSNS 

JWG 

LCSMM 

HAA 


US  Army  Materiel  Command 

Army  Materiel  Systems  Analysis  Activity 

Army  Training  and  Evaluation  Program 

Acquisition  Strategy 

Army  Streamlined  Acquisition  Process 

Army  Systems  Acquisition  Review  Council 

Battlefield  Development  Plan 

Combat  Developer 

Continuous  Evaluation 

Concept  Evaluation  Program 

Cost  and  Training  Effectiveness  Analysis 

Decision  Coordinating  Paper 

Director,  Operational  Test  and  Evaluation 

Defense  Systems  Acquisition  Review  Council 

Developmental  Training  Effectiveness  Analysis 

Detailed  Test  Plans 

Embedded  Training 

Early  User  Test  and  Experimentation 

Force  Development  Test  and  Experimentation 

Follow-on  Operational  Test  and  Evaluation 

Instructor  and  Key  Personnel 

In  accv.  'dance  with 

Independent  Evaluation  Plan 

Independent  Evaluation  Report(s) 

Initial  Operational  Capability 
Initial  Operational  Evaluation 
In-Process  Review 
Initial  User  Test  and  Evaluation 
Joint  Development  Test 

Justification  for  Major  System  New  Start 
Joint  Working  Group 
Life  Cycle  Systems  Management  Model 
Mission  Area  Analysis 
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LIST  OF  ABBEIEVIATIONS  AND  ACRONYMS  (Continued) 


MAATAG 

Mission  Area  Analysis  Test  Advisory  Group 

MADP 

Mission  Area  Development  Plan 

MADP 

Materiel  Acquisition  Decision  Process 

MANPRINT 

MANpower  and  PeRsonnel  INTegration 

MARB 

Materiel  Acquisition  Review  Board 

MATDEV 

Materiel  Developer 

MEP 

Master  Evaluation  Plan 

MOS 

Military  Occupational  Specialty 

MSC 

Materiel  Systems  Command 

NDI 

Non-Development  Item 

O&O 

Operational  and  Organizational  Plan 

OT 

Operational  Test 

OTEA 

Operational  Test  and  Evaluation  Agency 

OTP 

Outline  Test  Plan 

OTRR 

Operational  Test  Readiness  Review 

OTRS 

Operational  Test  Readiness  Statements 

P3I 

Preplanned  Product  Improvement 

PDM 

Program  Decision  Memorandum 

PFTEA 

Post-Fielding  Training  Effectiveness  Analysis 

PIP 

Product  Improvement  Proposal  (Program) 

PPQT 

Preproduction  Qualification  Test 

PQT 

Production  Qualification  Test 

PTEA 

Preliminary  Training  Effectiveness  Analysis 

QT 

Qualification  Test 

RAM 

Reliability,  Availability,  and  Maintainability 

RFP 

Request  for  Proposals 

ROC 

Required  Operational  Capability 

SAT 

Systems  Approach  to  Training 

SCP 

System  Concept  Paper 

SME,  SMEs 

Subject  Matter  Expert(s) 

SSP 

System  Support  Package 

SWG 

Special  Working  Group 

T&E 

Test  and  Evaluation 

C-2 


LIST  OF  ABBREVIATIONS  AND  ACRONYMS  (Continued) 


TDP 

TDS 

TDR,  TDRs 

TDY 

TEA 

TECOM 

TEMP 

TFT 

TIE 

TIR 

TISC 

TIWG 

TMEC 

TR 

TRADOC 

TSARC 

TT 

TT&E 

TTRR 

UT 

UT&E 

UTRR 


Test  Design  Plan 

Training  Development  Study 

Training  Device  Requirement (s) 

Temporary  Duty 

Training  Effectiveness  Analysis 
US  Army  Test  and  Evaluation  Command 
Test  and  Evaluation  Master  Plan 
Technical  Feasibility  Test 
Technical  Independent  Evaluator 
Test  Incident  Report 

Technology  Integration  Steering  Committee 
Test  Integration  Working  Group 
TRADOC  Materiel  Evaluation  Committee 
Test  Report 

US  Army  Training  and  Doctrine  Command 
Test  Schedule  and  Review  Committee 
Technical  Test 

Technical  Test  and  Evaluation 
Technical  Test .  Readiness  Review 
User  Test 

User  Test  and  Evaluation 
User  Test  Readiness  Review 


